garbage-collect doesn't release much space

[StanleyShen]

Hello, I am having a private docker registry, and we daily pushed images there, I had an job to run garbage-collect periodically, but I found the disk usage still keep increasing.

For example I have one image which is about 1.8G.
Before running the garbage-collect the disk usage is

/dev/xvdf1       99G   87G  6.7G  93% /var/vcap/store

After running garbage-collect, the disk usage is

/dev/xvdf1       99G   87G  7.2G  93% /var/vcap/store

Which means after deleting the image, it only releases about 0.5G disk space, so what's the remaining 1.2G about?
The registry version is ./registry github.com/docker/distribution v2.4.1+unknown

When running the garbage-collect I do see below logs like

INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/03/032c9cf1156abb7b4874b8e4023cf4914344d563cacae591dcf883035451f9bb  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/52/526200adb4e562d78fc42fa8db455f82e7ba7cfbb6ca206b9819b1d8cae1b20c  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/56/56652bc41678c3a0c39d71a7b5abfff52424dddcc9b5e10e6ab77f704d54fcdf  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/99/99647b2457ff2c14d0021fdda1136803b25d57d008957bdd5a18cd629c5b4e85  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/ce/ce7242dfaaec621f029795149ab0d6cb5467c5dc147680a21c6bda0064bb4a00  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/e3/e3a3b148f89d3c5ec415b0310936ad88afd814b5da2507a8c2e898d779f0174d  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/10/103dacd4981428e62a92586e434f31f021a84de6a3906e6cfca9b1f2c8bb77df  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/11/116679ed92079caa183bc74a6300b514b29ac5d6a086fbc9a1c3d1fb18d8f0f2  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/17/179bd293d2dfc0c039891e50ac056339f493fcca0b0fb4795fb67ef6ffe86efc  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/4f/4f4862a850ac45e9eca81b6dc4ca1cedc91c79e5316f125b4bb4b0b14e47d1af  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/89/8910ed079727b2440fe714e973181c624e688d09724b93f3a6576d4f426ff8c1  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/ad/adcaecf0c3a2c74b7d90a9b22ed8be10d2486723e6c4686430ac16381efbebe0  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/b1/b16bf7888d5ffeefad15ec2a811f247e65b6c75d4f79fa21acf14ce187594755  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/d9/d95803f917b8505a6a0a4c98c4f8710381dfa0b1f0c4f372ba20906c5f9c9995  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/18/18aefb43df71fc073c41a04689db8132ca5d803a50e8567ee5fff0297015e564  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/19/1983d3edccb0aead4e3fb3f7e5d8f38e85f8b74b8bcad1981fc3380c2e04478f  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/8a/8a13effafcb0396b401a116d0553f2ceeaedf0b934ee838a7c99cafdf9804fa0  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/92/9226016f37365aae77f30e37d47aa9d5934f7a8204eeceeef5704ef0dd9f3dd9  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/b3/b3741efd03d4535eeb167fdd6a7765aeca9dfcbd489a8a631da7177ae9b3ebc4  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/d4/d4e191bd4283c8ddc4cc57d9edf987a5cf773fb59591d993a54802f8ed59467c  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/5b/5b6df681cb1f4d11e6d1acf08d7dcfbed4c20bbcfec916f189b0946959332548  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/60/603a6affdfcb6943bc53535db3a3ddf376a1c7d7289c8418827b7e3ebc9a1c67  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/64/64e73dd0f364efef29b9abea57b59003d08b2317a8b1e440b3802db46b5c93dc  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/ca/ca0ec1eac59887dc3438fc175fc9330258764d64016fc983d36866119832f7aa  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry

Is there any known issue about this?

[mortensteenrasmussen]

Please note that Docker images are layers, and therefore deleting one image and garbage collecting will only delete the layers not used anymore.

consider this image:

alpine
java
your-jar-file.jar

If you have this image in tag 0.1 and 0.2 and you delete 0.1, the garbage collect will only delete the last layer with your jar file (because the other layers are also used by 0.2.

For excessive disk usage, you might be running into another gotcha, which is that if you push latest, and push latest again later (with different code), the old latest manifest is still valid (and can be fetched by digest), and can therefore not be garbage collected. I'm working on a script to clean up these manifests without any tags; I'll let you know when I have something.

A feature request to be able to garbage collect this is #1844.

[StanleyShen]

Thanks for information.
For the image I post above is just created recently and I just used latest tag, but I deleted the whole image, maybe some layer are used by other images also.
For other images, I used different tags each time, but even in this case, it seems there are still some layers are left after deleting some old tags.

I once decided to keep latest 6 tags for each image, at that time even we have 6 tags for each image , we still had some free disk space there.
Recently, it warns us the disk usage is almost 100% now, and I changed to keep latest 4 tags for each image, it freed some space, but after some days, the disk is almost 100% again.

[mortensteenrasmussen]

Hi, it sounds like you might have a lot of untagged manifests. To verify this, could you please run this on your registry server in the $REGISTRY_DIR/data/docker/registry/v2/repositories folder.

It will return the number of manifests you have in your registry that doesn't have a tag associated.

comm -23 <(find . -type f -name "link" | grep "_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/ '{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep "_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g'; echo; done | sort) | wc -l

The command looks cryptic, but it uses the filesystem to check tags vs. manifests and returns the manifests that don't match any tags. If this number is high, I can help you with a command that can delete them. I'm planning to put it on Github, but I'm still testing/verifying it.

[antoinetran]

Hi, we have the exact same issue. We followed the procedure the official doc, but end up still having full space. In our case, we have images that has no tag at all, and it still takes lots of space. Only when we delete the hole image (that contains no tag), then we can garbage-collect a lot of space.

Here is the result of the command:

comm -23 <(find . -type f -name "link" | grep "_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/ '{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep "_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g'; echo; done | sort) | wc -l
299

I was suspicious the manifest still has a lots of references to unused layers, even when we delete tags.

[mortensteenrasmussen]

Hi, this means you have 299 manifests (docker images) that have no tags. If you don't use pull-by-digest but only pull by image:tag, then you could delete them.

This gist is a script to do that. Back up your registry beforehand. Please note that I'm not liable for anything that happens and all that stuff :)

Replace the path to your registry, as well as the ip/port of your registry.

It uses the same command like the one above, and then issues DELETE's to your registry for the manifests found. In my testing, it worked without any problems, and all my actual images and tags worked fine afterward. I cleaned up 200gb of a 255gb repo.

[antoinetran]

It did removed 304 manifests file. However, the garbage-collect reclaimed nothing. I have to delete _manifests folder to reclaim spaces.

[mortensteenrasmussen]

Did you run a registry garbage collect after running the script?

[antoinetran]

Yes. But if I re-run the script, it stills deletes 298 manifests file each time. So I guess the delete part does not work.

[mortensteenrasmussen]

So if you run the first command again, it should show close to 0. Correct?

If not, then I think something is wrong with the requests to the registry (the curl command). Please check, and if the number is the same as before you ran the cleanup, please try to run one of the curl commands manually to figure out what's going wrong.

Also, please move this discussion to the gist, it's cluttering up the issue.

[antoinetran]

Ok found it, firstly, we used https instead of http, so it did http://https://.. . Secondly, I added --insecure in curl, because we use auto-generated certificate for now.
We reclaimed ~40GB of space, that's cool!

Somehow, this procedure should be in the official doc one day, because the workflow of overwriting latest layers in very used in every development environment.

Thank you again.

[mortensteenrasmussen]

You're most welcome. Glad I could help.

I updated the gist to use registry URL instead of host and port variables. Should help people later.

[StanleyShen]

@mortensteenrasmussen I run

comm -23 <(find . -type f -name "link" | grep "_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/ '{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep "_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g'; echo; done | sort) | wc -l

In my docker registry VM, and it shows 751

[mortensteenrasmussen]

@StanleyShen Please try to run the gist I linked above

Read the other comments if you use https for your repo. It should delete the 751 docker images you have that doesn't have any tags.

If that works, please mention how much space it freed up, and then we can close this issue. I'm working on fixing up the cleanup script and put it in a docker container that's easy to use, and I'll link to the repo here when ready.

[antoinetran]

Do you think it is a good idea to submit a feature request about the registry v2 API? This script just shows the registry is missing an official way to clean untagged image, the same way we clean docker cache using "--filter dangling=true". This feature has cost us days, because it forces us to delete all image tags to reclaim spaces. If yes, would an issue in github be the best way for such a request?