Broker randomly is unable to connect to etcd svc

[vatsalparekh]

Bug:

What happened:
I am seeing some apbs unexpectedly gone, previously I saw all the apbs, but after sometime they disappear, and after sometime again they come back.

Some logs from the apb pod

[2018-05-11T07:58:24.69Z] [ERROR] - Something went real bad trying to delete batch specs... - client: endpoint https://asb-etcd.ansible-service-broker.svc:2379 exceeded header timeout
[2018-05-11T07:58:24.69Z] [ERROR] - Failed to bootstrap
[2018-05-11T07:58:24.69Z] [ERROR] - client: endpoint https://asb-etcd.ansible-service-broker.svc:2379 exceeded header timeout

After this log mostly, some apb disappear for a while
What you expected to happen:
No to disappear randomly
How to reproduce it:
Not reproducible in a pattern, but occurs randomly from time to time

[eriknelson]

Discussed at the 05/15 broker community meeting. Action was to update the title to reflect the underlying problem. The failed connection will manifest in missing APBs, and it sounds like a bootstrap will kick in again and be able to connect to the service.

Fundamentally this looks like a networking problem in the environment. It's hard to deduce what is causing this unless there's something reproducible.

[rthallisey]

I think this is related to #876

Workaround is:

oc annotate route -n ansible-service-broker asb-etcd --overwrite haproxy.router.openshift.io/timeout=300s

Closing this and we'll continue to track in #876. Feel free to reopen if you disagree.