Old containers are not stopped after deployment

[janericwinkelmann]

Hi there,

for about a week we have a major issue with our caprover server. It was running fine for a year now and suddenly this happened:

One morning we have seen that a lot of our apps where not running. So we logged into SSH and inspected the containers as well as the services. A lot of the services where at scaling 0/1 so we tried to scale them back to 1/1. For the majority of apps this worked, in others it didn't because some images where missing. For these apps we had to do a full redeploy. We have them all up and running again but facing a new issues that occurs from time to time:

When we deploy an app update the scaling of the service sometimes goes up to 2...3...4/1. It seems as if the old containers are not being killed. This results in the fact that nginx sometimes directs to the old version of our app and sometimes to the new one. When we notice this, we have to login to the server via SSH and downscale the app. That's working without an issue and everything seems to be fine after that. Until the next deployment.

Do you have any idea what might cause this? It might be a docker issue but we are not sure what to look for. Any help would be much appreciated! Thanks guys

[githubsaturn]

This is a Docker issue, have a look at these ones:
moby/moby#24394
moby/swarmkit#1372
moby/moby#24670

Previously it's been reported in CapRover a few times, but there is generally nothing that CapRover can do to fix/mitigate the issue:
#784
#896
#1074

[githubsaturn]

I suggest you open a new issue on Docker's github page with your specific data, specially when it goes beyond the desired count, e.g., 3/1, this should never happen. Not sure why it hasn't fixed properly in Docker yet...