With too many apb's bootstrap causes the route to timeout.

[jmontleon]

Bug:

Pretty low priority at this point.

With too many apb's bootstrap causes the route to timeout.

What happened:

apb bootstrap
Contacting the ansible-service-broker at: https://asb-1338-ansible-service-broker.172.17.0.1.nip.io/ansible-service-broker/v2/bootstrap
Error: Attempt to bootstrap Broker returned status: 504
Unable to bootstrap Ansible Service Broker.

What you expected to happen:
Boostrap completes.

How to reproduce it:
Populate a whole lot of APB's. In my cases I created 150 service templates in CFME while running a simple test of the adapter. Those were in addition to the ones available at docker.io/ansibleplaybookbundle that were also being loaded.

Boostrap actually completes but the route times out. I'd guess the defaults is ~10-30 seconds. My boostrap is taking probably closer to a minute

oc annotate route -n ansible-service-broker asb-1338 --overwrite haproxy.router.openshift.io/timeout=300s stops it from happening.

We could probably add the annotation to the route we create with a reasonable setting higher than the default, otherwise as we see users with more than a handful of APB's we may start seeing folks reporting this.

[SaravanaStorageNetwork]

@rthallisey @jmontleon Thanks! It helped - able to run bootstrap only after extending the timeout as mentioned.

[jwmatthews]

Assuming correct fix here is to make this an async call, thoughts?
If we wanted to make this async assume we align this to 3.11 to limit churn in 3.10.

[jmrodri]

@jwmatthews yes if the bootstrap is taking too long, we should spawn it off as we start.

[jmontleon]

@jwmatthews @jmrodri @dymurray when running apb boostrap it's waiting for the bootstrap to complete to also do a relist so the catalog sees the updates as well.

If it was changed to async would that behavior have to be changed in the client tool?

[jmrodri]

@jmontleon I misread the original complaint about apb bootstrap. I thought it was during broker startup. hrm that makes it a bit more complicated. Making it async would mean the bootstrap call returns immediately but we'd need a way to get back the same information.

[jmontleon]

We could probably annotate the route we create with a 1 or 2 minute timeout for now. I think it would cover most cases for the time being. If we ever see cases of someone with 100's of APB's we might have to readdress it, but even close to 200 was returning for me in under a minute.

[jwmatthews]

Example @jmontleon shared:

oc annotate route -n ansible-service-broker asb-1338 --overwrite haproxy.router.openshift.io/timeout=300s

[jmrodri]

Fixed by PR #1008

[wenchma]

@jwmatthews it's unlucky, the 504 error not gone after increasing the timeout as your mentioning .
😭

# svcat get brokers
           NAME                                                        URL                                              STATUS  
+-------------------------+-------------------------------------------------------------------------------------------+--------+
  ansible-service-broker    https://automation-broker.automation-broker.svc:1338/automation-broker/                     Ready   
  template-service-broker   https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io   Ready   

# apb push --broker https://automation-broker.automation-broker.svc:1338/automation-broker
version: 1.0
name: my-new-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: my-new
  dependencies: []
plans:
  - name: default
    description: This default plan deploys my-new-apb
    free: True
    metadata: {}
    parameters: []
Found registry IP at: 172.30.138.73:5000
Finished writing dockerfile.
Building APB using tag: [172.30.138.73:5000/openshift/my-new-apb]
Successfully built APB image: 172.30.138.73:5000/openshift/my-new-apb
Found image: docker-registry.default.svc:5000/openshift/my-new-apb
Warning: Tagged image registry prefix doesn't match. Deleting anyway. Given: 172.30.138.73:5000; Found: docker-registry.default.svc:5000
Successfully deleted sha256:906da7a2ddb5cbb925f2cc27b9199d396c370c81322a36c5cfe774985b70718f
Pushing the image, this could take a minute...
Successfully pushed image: 172.30.138.73:5000/openshift/my-new-apb
Contacting the ansible-service-broker at: https://automation-broker.automation-broker.svc:1338/automation-broker/v2/bootstrap
Error: Attempt to bootstrap Broker returned status: 504
Unable to bootstrap Ansible Service Broker.

[jmontleon]

There are a couple PR's hanging out to fix this. Trying to get them merged:
automationbroker/automation-broker-apb#27
#1033

@wenchma can you describe the asb route and verify the annotation is there? Is it actually taking 5 minutes (300s) to return now if you set that value? If that doesn't seem right maybe we're hitting something else.

[wenchma]

@jmontleon thanks for your reply, I guess I need a longer timeout.

[jmontleon]

As mentioned it there might be something else happening that we haven't run into before as well. 5 minutes seems like an awfully long time to bootstrap unless you are loading a huge number of APB's. I suppose it is possible, but managed to get to only around around 60s by creating a couple hundred APB's.

[wenchma]

@jmontleon it looks like that the 5 minutes timeout did not take effect. it was obvious it returned 504 error in less than 5 minutes.

[jmrodri]

Closing as this has been fixed.