unpredictable 503 Service Unavailable with multiple instances of DFP

[lgazo]

Description

I have DFP deployed as Docker Swarm stack 2 replicas across 2 different nodes. In front of DFP there is a loadbalancer. At the moment LB points to just one node in order to reduce the debugging surface. For the sake of the test I have also passed LB and executed requests directly against DFP and found out it does not have impact.

Steps to reproduce the issue:

1. Deploy DFP ... wait
2. Deploy application stack on application node 1
3. Wait, to give cluster the chance to not have excuses
4. Repetitively execute requests against DFP
5. Deploy another version of stack on application node 2
6. Wait, to give cluster the chance to not have excuses
7. Repetitively execute requests against DFP

Describe the results you received:

Usually when first app stack is deployed DFP does not return 503. But almost always it does randomly when second stack is deployed.

I have also checked /cfg/haproxy.cfg on both proxy nodes but they are the same. I have also tried DO_NOT_RESOLVE_ADDR=true but without effect.

Once I decrease the number of replicas to 1, everything is stable.

In the proxy log on the node where the request reaches I get something like this:

2018/06/09 15:43:11 HAPRoxy: 10.255.0.9:43070 [09/Jun/2018:15:43:11.898] services~ services/<NOSRV> -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET /console/ HTTP/1.1"

... although the service (the container is up & listening, the entry is in haproxy.cfg) is up.

Describe the results you expected:

I expect no 503 for replicas > 1 as well.

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

DFP tested: 18.05.26-51 and previously also 17.09.22-18
Docker engine: 17.09.0-ce

[thomasjpfan]

@lgazo There were similar issues that were reported and fixed in the more recent versions of DFSL. Can you try dockerflow/docker-flow-swarm-listener:18.06.03-26 and see if it fixes your problem

If that doesn't resolve your issue. Can you provide the following information?

1. Your stack files. In place of your application you can use https://hub.docker.com/_/nginx/ as a placeholder with com.df.servicePath=/.

2. The haproxy.cfg file before and after the issue appears.

[lgazo]

@thomasjpfan I use newest DFSL, the version you mentioned.

Proxy stack file is here:
single-proxy.stack.compose.v3.yml.txt

Application stack (simplified) here:
stack.compose.yml.txt

[thomasjpfan]

@lgazo

1. This can be a docker bug. Can you test out your stack on 18.03.1-ce?

2. For testing purposes, can you simplify your stack and just use plain HTTP (no letsencrypt)? This would help narrow down the cause of the bug.

[lgazo]

@thomasjpfan I have upgraded edge proxy nodes to 18.03.1-ce, it did not help. I have upgraded also manager node where DFSL runs, no effect. The only effect now is that previously when 503 appeared, it was related always to one edge node (I can see in logs where the request ends), now after upgrade it changes in time on the same node.

I will try to deploy different stack with HTTP and nginx and try it there.

[lgazo]

I have upgraded also whole swarm. The end results are two:

• I had to remove "ingress" network from edge proxy stack because of following error: Error response from daemon: rpc error: code = InvalidArgument desc = Service cannot be explicitly attached t o the ingress network "ingress". I thought that this network might cause problems, but it is not the case
• now even when I limit proxy to one node it does not work ... which is very unfortunate, and I guess I cannot go back

[thomasjpfan]

1. It would be best to do this type of testing in a staging environment.

2. I want to make sure your stack works without nib0r/docker-flow-proxy-letsencrypt. Are you able to run your stack without SSL?

[lgazo]

I have skipped nib0r/docker-flow-proxy-letsencrypt but no change. I called What I have noticed is that it takes couple of minutes until it starts resolving service (it does not show 503) but after a while it does not again. It does not make sense at all because I see the config in haproxy.cfg, I can call the service through wget. What else can be wrong in the configuration? Is there a way to debug or determine why it resolves to 503?

[lgazo]

and one observation, clue hopefully... when I start proxy stack, it has just one process:

/ # ps aux
PID   USER     TIME   COMMAND
    1 root       0:00 /sbin/tini -g -- docker-flow-proxy server
    5 root       0:00 docker-flow-proxy server
   19 root       0:00 ash
  120 root       0:01 haproxy -f /cfg/haproxy.cfg -D -p /var/run/haproxy.pid -x /var/run/haproxy.sock -sf 117 
  371 root       0:00 ps aux

once I redeploy application stack, there are multiple processes

/ # ps aux
PID   USER     TIME   COMMAND
    1 root       0:00 /sbin/tini -g -- docker-flow-proxy server
    5 root       0:01 docker-flow-proxy server
   19 root       0:00 ash
  440 root       0:00 haproxy -f /cfg/haproxy.cfg -D -p /var/run/haproxy.pid -x /var/run/haproxy.sock -sf 120 
  448 root       0:00 haproxy -f /cfg/haproxy.cfg -D -p /var/run/haproxy.pid -x /var/run/haproxy.sock -sf 120 
  690 root       0:00 haproxy -f /cfg/haproxy.cfg -D -p /var/run/haproxy.pid -x /var/run/haproxy.sock -sf 680 
  705 root       0:00 ps aux

Couldn't it be the problem that only one of those processes has most actual configuration and others are obsolete?

Is there a way how to force haproxy to refresh those processes (even manually from inside the container) or purge them?

[thomasjpfan]

Does force restarting DFP remove the other processes and resolve the 503 issue? You can do this by running:

docker service update --force <NAME_OF_DFP_SERVICE>

This should force DFP to have only one process. If this solves your issue, we can confirm that the multiple process is the root of the problem.

[lgazo]

Yes I do confirm that now. Once there is fresh state with just one process, it behaves correctly with one replica and two as well.

[thomasjpfan]

Thanks for the information. I will work on a fix for this issue.

[thomasjpfan]

@lgazo I issued a fix that tries to address your issue. Can you test out your setup with dockerflow/docker-flow-proxy:18.06.11-52?

[lgazo]

I have tested it but not a big difference... it seems more stable but still fails + there are multiple processes, number seems increasing by number of redeploys of stack

[thomasjpfan]

@lgazo It seems like you are able to recreate this issue really easily. Can you provide docker commands and stack files to recreate this issue using docker-machine? I am having trouble reproducing this issue.