Confirm haproxy state infromation from DFSL before starting the DFP container

[jwalkins2017]

Description
I am using DFP and DFSL together, if the DFP and DFSL container is on the same docker node and the node needs to be taking down for maintenance, the DFP container will general startup before the DFSL, causing 503 to be sent until the DFSL is available.

Steps to reproduce the issue:

1. Establish a 2 or 3 node docker swarm cluster
2. Set the replica for DFP service to match the number of nodes in the cluster
3. Set the node that hosts the DFSL and one of the DFP to drain, while generating http requests

Describe the results you received:
When the DFP moves to another node, it will generate 503's until the DFSL is online

Describe the results you expected:
If the DFP service is configured to use a DFSL, do not start the DFP until the container can gather the correct haproxy cfg from the DFSL.

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

[thomasjpfan]

This is an interesting feature request. Currently DFP considers itself to be healthy as long as it can start up its service independent of DFSL. I think keeping this behavior for backwards compatibility is important.

I propose adding an env variable named HEALTHCHECK_INCLUDES_RELOAD which will add an additional healthcheck that ties DFP with DFSL. DFP will only be considered healthy if DFSL is up and DFP was able to reload with service information from DFSL.

[thomasjpfan]

@vfarcic What is your opinion of the name HEALTHCHECK_INCLUDES_RELOAD for this feature flag? When this flag is true, the healthcheck will include three more items:

1. Make sure LISTENER_ADDRESS is not the empty string.
2. Ping DFSL to make sure it is up.
3. Send a reload request to DFP. This way DFP is considered healthy when it was able to fetch services from DFSL and reconfigures haproxy.

[vfarcic]

Maybe it would make sense to include the additional checks if LISTENER_ADDRESS is not empty without introducing more vars. I don't think that would break compatibility since we can assume that those with that variable do have DFSL running. What do you think @thomasjpfan ?

[thomasjpfan]

@vfarcic I like that idea. I have updated DFP to reflect this change.

@jwalkins2017 Please test out dockerflow/docker-flow-proxy:18.06.27-64 and see if it works for your use case.

[jwalkins2017]

I will be testing this tomorrow (6/29). Thank you for the quick turn around and for putting the project together.

[thomasjpfan]

I fixed a bug introduced by this feature. The fix is at dockerflow/docker-flow-proxy:18.06.29-65.

Your issue may not be resolved yet.

[thomasjpfan]

I reintroduced the HEALTHCHECK without the bug. Please test it out at dockerflow/docker-flow-proxy:18.06.29-66.

[jwalkins2017]

I just tested 18.06.29-66, and everything looks good. I continue receiving HTTP 200 responses, while the SL moves to a new manager and new DFP are being spun up.

[thomasjpfan]

I made a slight tweak to the healthcheck.

Before, the healthcheck will check once, and quickly fail after seeing it has not initially reloaded. I thought this would not be good for situations where DFP needs a little more time to initialize. I updated the healthcheck to check every 500 ms for the initial reload.

Please test this update at dockerflow/docker-flow-proxy:18.07.03-68 and see if it works for you.

[jwalkins2017]

Everything is still working as expected. Thank you