Be aggressive about removing all intermediate containers #478
Conversation
Autumn has been having space issues (and has been running out of inodes). It's not clear what is causing this, since docker system prune doesn't seem to help. We already run `docker run` with `--rm` argument. This commit changes the build also to remove intermediate containers in case of build failures.
|
Interesting :/ I poked around and ran more docker pruning (which didn't help much), but there's something I don't understand with the docker overlay2 directories: $ df --inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_boot-data 13033472 11697303 1336169 90% /data
^^^^^^^^
$ ls /data/docker/overlay2 | wc -l
376It looks like the Why not, but some of those directories are a bit old: And I can't find why docker retains them after all of our system pruning: Any idea? I don't want to remove them manually as I don't understand why they are there :P Perhaps we could stop all of our dockers, run the system prune, and see if anything remains? |
It may be worth trying, but the service will be down for a while, before all the @mtelvers and I tried to remove all of the overlay2 directories that seemed unused, but that caused build failures. But, later I moved out a handful of them using dates, and it did create some space for redeploying the production instance. There's a thread on #ci-dev that I'll DM you on slack, which might add more information. |
|
I have setup my own current bench to try to understand what is happening. Building the system creates around 18GB of Docker images. Then for each run of the test project there seem to be three directories left over in the docker/overlay2 folder. Of interest, I note that |
|
Having experimented some more on my installation, I propose the following course of action:
If this proves successful, we can move this installation on to the SSD. If it fails, edit |
|
See also ocaml/infrastructure#70 |
I've followed the steps proposed by you, and the docker service is now using the spinning disk. If everything works fine, overnight, we could plan on moving things to the SSD. |
|
The docker service now uses the SSD. |
Autumn has been having space issues (and has been running out of inodes). It's not clear what is causing this, since docker system prune doesn't seem to help. We already run
docker runwith--rmargument. This commit changes the build also to remove intermediate containers in case of build failures.