Zero Downtime Deployments

[tpfwrz]

Is your feature request related to a problem? Please describe.
Zero downtime deployments are a must for long running applications that always have connected users.
I am a big fan of cap rover and this is it's only lacking area in my opinion.

Describe the solution you'd like
Almost all deployment systems have zero downtime deployments, even dokku does.
What are the technical limitations to making this a reality for caprover?
Obviously in your head it's as easy as keeping the previous replica alive till new one is active, or having multiple replicas and then removing the old ones when deployment is active, but since this feature has not existed for years now, it for sure is a bit more complicated than that.
I would be happy to contribute development to this feature if I can.

Describe alternatives you've considered
Not sure.
A clear and concise description of any alternative solutions or features you've considered.

Additional context
This would really make caprover a lot better and more salable.

[githubsaturn]

EDIT (2024): 🎊
A new article is now added to the documentation to help better understand this issue:
https://caprover.com/docs/zero-downtime.html

Old response:

TLDR:
CapRover does support zero downtime (mostly). What makes you think that CapRover doesn't have zero downtime?

More details:
If your app doesn't use a volume, CapRover uses start-first strategy when updating the containers. This means that the newer version of your container will be up and running before the old one is killed. This should give you next-to-zero downtime. There is still a slight chance that your container wakes up but internally it takes some time before it can accept requests. This is out of CapRover's hands.

This strategy is intentionally not applied to apps with volumes attached to them. This is because if multiple instances of the same service are trying to access the same file, this will result in data corruption and failures. For apps with volumes (persistent data), CapRover user stop-first strategy. This means that the old container is stopped first, before the new container starts up. This results in a few seconds of downtime.

[slava-vishnyakov]

EDIT (2024):

Make sure you don't have "persistent data" enabled on your app, then add to Dockerfile:

HEALTHCHECK --interval=1s --timeout=1s --start-period=1s --retries=30 \
  CMD [ "/bin/sh", "-c", "if [ -f /tmp/started ]; then exit 0; else curl -f http://localhost/ | grep 'ALIVE' && touch /tmp/started || exit 1; fi" ]

This tries to connect to http://localhost/ and if it sees a certain string (ALIVE in the example), it stops the checks. It will try to repeat the check 30 times with an interval of 1 second. If successful, it will create /tmp/started file and won't hammer http://localhost/ anymore

---

There is still a slight chance that your container wakes up but internally it takes some time before it can accept requests.

For us, it's 100% of the time and users end up seeing 502 CapRover screen instead of the app. But it wasn't this way when we used Dokku. We like CapRover way more, but absence of real zero-downtime deployments is a real mood killer.

This is out of CapRover's hands.

Actually, it's not quite that. Dokku has it.

If you look at Dokku https://dokku.com/docs/deployment/zero-downtime-deploys/
scroll down to "Check instructions", basically, you can create a simple file called CHECKS file, like this:

/about  Our Amazing Team
/stylesheets/index.css  .body

After a deploy and starting the contained Dokku will wait for DOKKU_CHECKS_WAIT (default: 5) seconds, then try to curl http://{container-ip}:{active-port}/about and see if Our Amazing Team is there, if it is it will proceed to curl http://{container-ip}:{active-port}/stylesheets/index.css and check if .body (string) is there. If it is, it will switch nginx from the old container to the new one. If it doesn't find these strings within DOKKU_CHECKS_TIMEOUT (default: 30) seconds, it will restart tests up to DOKKU_CHECKS_ATTEMPTS (default: 5) times, and if they fail, it will mark deploy as failed and reject it.

These also act as smoke tests and saved us numerous times from bad deploys. But they also ensure that containers are really ready to accept traffic. I can probably devote some time to try to implement it in CapRover, but maybe you can give me some hints as how this could be implemented?

[githubsaturn]

@slava-vishnyakov - there is a different implementation across Dokku and CapRover. Dokku has to manage the lifecycle of containers itself since it doesn't use an orchestration system as it was created way before Docker swarm. CapRover on the other hand, uses Docker swarm.

But that doesn't mean that you should see 502 page during deploys. Docker swarm is supposed to handle all checks automatically through HEALTHCHECK.

• Is your app stateless or does it have any mounted volumes?
• Are you using a specific HEALTHCHECK in your dockerfile?

[frederikhors]

My deploy happen every hour or so during the day with no issue at all with Caprover!

[slava-vishnyakov]

there is a different implementation across Dokku and CapRover

@githubsaturn I understand, thank you!

• Is your app stateless or does it have any mounted volumes?

Stateless.

• Are you using a specific HEALTHCHECK in your dockerfile?

We used to have HEALTHCHECKs, but the problem is that they aren't "deploy" checks, they are "healthchecks", so they keep banging our container non-stop. And if there's a temporary hiccup, it starts a cascade of restarts (in our case). I guess we have no option but to implement some kind of healthcheck that would simulate a deploy check.

The bad part about this is that we're both stateless and we don't use Docker Swarm, so a deploy check would be really helpful here. (Though, I think that Dokku even does zero-downtime deploys for stateful containers)

Anyway, thank you for taking the time to explain it.

@frederikhors There are no problems with deploys on caprover, the problem is that our container takes 5-30 seconds before it's really "responsive" due to how it works, but CapRover makes it "live" immediately. So, the users see 502 for quite some time.

[slava-vishnyakov]

Oh and I forgot to mention one more benefit that Dokku's zero-downtime deploys have over HEALTHCHECKS in CapRover. If you deploy and your HEALTHCHECK for some reason fails, you see that you deployed successful, but you actually see the old version (so, in a sense, your deploy failed) without any notification that your deploy failed.

I'm not sure how to solve this in CapRover (with the Swarm and all), so I'm just thinking out loud.

[githubsaturn]

We used to have HEALTHCHECKs, but the problem is that they aren't "deploy" checks, they are "healthchecks", so they keep banging our container non-stop. And if there's a temporary hiccup, it starts a cascade of restarts (in our case). I guess we have no option but to implement some kind of healthcheck that would simulate a deploy check.

The HEALTHCHECKs are supposed to be very light and just really report the health of the application. You can define delays, attempts etc to match the behavior you need.

Search for "health" to see all tweaks that you can make to your service to make healthcheck work for your use case.

I will be very surprised if there is a common usecase that is not covered by Docker's HEALTHCHECKs.

[smyja]

There is still a slight chance that your container wakes up but internally it takes some time before it can accept requests.

For us, it's 100% of the time and users end up seeing 502 CapRover screen instead of the app. But it wasn't this way when we used Dokku. We like CapRover way more, but absence of real zero-downtime deployments is a real mood killer.

This is out of CapRover's hands.

Actually, it's not quite that. Dokku has it.

If you look at Dokku https://dokku.com/docs/deployment/zero-downtime-deploys/ scroll down to "Check instructions", basically, you can create a simple file called CHECKS file, like this:

/about  Our Amazing Team
/stylesheets/index.css  .body

After a deploy and starting the contained Dokku will wait for DOKKU_CHECKS_WAIT (default: 5) seconds, then try to curl http://{container-ip}:{active-port}/about and see if Our Amazing Team is there, if it is it will proceed to curl http://{container-ip}:{active-port}/stylesheets/index.css and check if .body (string) is there. If it is, it will switch nginx from the old container to the new one. If it doesn't find these strings within DOKKU_CHECKS_TIMEOUT (default: 30) seconds, it will restart tests up to DOKKU_CHECKS_ATTEMPTS (default: 5) times, and if they fail, it will mark deploy as failed and reject it.

These also act as smoke tests and saved us numerous times from bad deploys. But they also ensure that containers are really ready to accept traffic. I can probably devote some time to try to implement it in CapRover, but maybe you can give me some hints as how this could be implemented?

i'm also experiencing this issue. There's no zero downtime deployments.

[activenode]

TLDR: CapRover does support zero downtime (mostly). What makes you think that CapRover doesn't have zero downtime?

More details: If you're app doesn't use a volume, CapRover uses start-first strategy when updating the containers. This means that the newer version of your container will be up and running before the old one is killed. This should give you next-to-zero downtime. There is still a slight chance that your container wakes up but internally it takes some time before it can accept requests. This is out of CapRover's hands.

This strategy is intentionally not applied to apps with volumes attached to them. This is because if multiple instances of the same service are trying to access the same file, this will result in data corruption and failures. For apps with volumes (persistent data), CapRover user stop-first strategy. This means that the old container is stopped first, before the new container starts up. This results in a few seconds of downtime.

I think this is a decision you should leave up to the developers as they know their apps best.

See, reason is: With this reasoning you could even argue the same thing without having a volume e.g. If I am using a static build that is using a database elsewhere. You cannot protect devs by assumptions really.

Like e.g. if my volume is used for cdn reasons and my CDN file names are controlled by a database, there is no reason not to let multiple containers access that volume at the same time.

So maybe go rather like having a checkmark:

"You have a volume attached, do you wanna have zerodowntime deployment with the following implications? ..."

[githubsaturn]

@activenode - you are certainly correct and the decision is definitely up to the users. These are simply the default behavior. You can override this (and many other defaults) by using service override.

In this case, simply add the following snippet to your service override:

UpdateConfig:
  Order: start-first

[activenode]

Wonderful, thanks @githubsaturn