mrsk deployments filling up disk without cleaning overlay2 directory
#403
Comments
|
|
|
running |
|
@danthegoodman1 - MRSK doesn't call It will run commands like these:
docker ps -q -a --filter label=service=app --filter status=created --filter status=exited --filter status=dead | tail -n +6 | while read container_id; do docker rm $container_id; done
docker image prune --force --filter label=service=app --filter dangling=true
docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' --filter label=service=app | tr -d '\n')registry:4443/app:latest\|registry:4443/app:<none>" | while read image tag; do docker rmi $tag; doneThis ensures we have just the last 5 containers and their images. These commands all are filtered by Please let me know if that's the case. Either way we should add a check for that label and reject images that are missing it. |
|
I am building them externally, perhaps they were ignored by the prune? I deploy like:
deploy.yaml: |
|
I'm also noticing that |
|
Seems to mostly be from one layer as well |
|
I am observing the same, our hosts are being filled with images:
Debugging a bit, I found kamal is using the following command: I found two issues with this:
Important note: we are building the docker images outside of Kamal and pushing them to DockerHub. Then deploying with For anyone using GitHub Actions, I fixed it like this: The image tagging is also "on us" because we are tagging each image with the GitHub commit. That said, I think Kamal should just prune all images that don't have a running container: If only one app is supposed to be running on a server at once, it makes sense that we only keep that container's image, right? @djmb would you be open to a PR changing this? For now I added a post-deploy step in our CD that runs: (I found the command digging on the code) |
|
Hi @yoelcabo! Thanks for the investigation! I think we should introduce a check for external images to ensure that they have the correct labels, which should prevent that in future. I'll get to this soon. Regarding When you deploy the image is tagged with the version and also The integration tests do a rollback as part of them so you could test this out there if you want to look into this in more details. |
and others
It seems that the filesystem of old images are retained, and thus disks fill indefinitely. I've got workers that I've deployed updated images to <100 times so it's mostly a small single layer change.
overlay2dir:The text was updated successfully, but these errors were encountered: