Repository navigation
Stop the old app container before removing it on an update #125
namespaceMarcello
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
On an update,
deployWithVolumestarts the new container, switches the proxy to it, and thenremoveContainersExceptremoves the old one withContainerRemove{Force: true}(internal/docker/application.go). Docker kills a running container with SIGKILL before it removes it, so the app in the old container never gets its stop signal, and whatever it's doing in the background stops wherever it is.HTTP requests are already covered:
kamal-proxy deploydrains the old target (its--drain-timeout, 30 s by default) before it returns, and ONCE only removes the old container after that. Background work isn't drained. In Campfire, for example, a job running in the old container when an update lands (a bot's webhook, the push notifications for a message) is cut off and doesn't run again.We measured it with Campfire, which stops its processes in order on SIGTERM once basecamp/once-campfire#401 is in. A bot whose webhook answers after 3 s, with the container stopped 1 s into the job:
docker stop(whatApplication.Stopdoes)docker kill(what the forced removal does first)(Docker Desktop on Windows 11, container limited to 2 CPUs.)
Our suggestion is to stop the old container before removing it, the way
Application.Stopalready does:ContainerStopOptions{}sends the image's stop signal, and Docker kills the container only if it's still running after its stop timeout (10 s by default). Traffic is already on the new container by then, so this adds no downtime; the update just takes up to that long to finish. A container that doesn't stop in time is still killed and removed, so an update can't hang on it.If this direction sounds right, we're happy to send it as a pull request with a test.
Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.
All reactions