Impossible to disk cleanup the registry (2.5.0) due to broken garbage cleanup?

[zoobab]

Hi,

I am trying to cleanup a 2.5.0 registry which is getting its main disk full.

I deleted some tags, run the garbage cleanup cron job, disk was freeed, but when I try to repush an image that was deleted, I hit the following error "Layer already exists".

I have also originally posted here:

docker-archive/docker-registry#988

While I think this is the appropriate place to mention the issue.

Can someone from Docker confirm the issue?

This is really an annoying bug, because it prevents the real "Continuous Integration" to happen.

[dmcgowan]

Can't really confirm anything without more information. It is possible that there was a still a tag or manifest which was referencing the layer, so seeing "Layer already exist" itself would not be indicative of an error.

Are you having any other symptoms? You mentioned that that disk space was freed, but perhaps not as much as anticipated and it is filling up again too quickly? In addition to deleting the tags, are you also deleting manifests which may be untagged?

[zoobab]

Steps to reproduce:

1. Delete an image tag
2. Run the garbage collector
3. Repush the same image+tag

Will try to give you my exact steps with curl next monday, but I think in the 988 ticket it is well explained.

[eesprit]

As seen in docker-archive/docker-registry#988 restarting the registry is needed and solves the problem.
I think the recommended way to do GC is to stop registry, GC, and then start it (to be sure that nothing is getting uploaded during GC).

It might be complicated when running the registry in a docker container though, as stopping the server would stop the container and not allow to exec the GC in it...
The good way is probably to restart the server in read-only mode, GC, restart in read-write mode.

[zoobab]

@eesprit by restarting the server in read-only mode, you mean creating a second config2.yml with this section as difference:

 maintenance:
    readonly:
      enabled: true

And then restarting the server with that readonly option, run GC, restart the server with the old config?

Or is there an API way to tell the running server to put itself in readonly mode?

[eesprit]

I don't think there is an API call for that (the Docker Private Registry expose one, but this does not seem to exist in the distribution registry).

So yes, I think the only way is to switch conf files.

By the way, as far as I understand, the maintenance/readonly is under storage, not at the "root" of the config file.

[dynek]

I am also able to reproduce this (exactly as zoobab says it).

registry github.com/docker/distribution v2.5.1

# curl -kIXDELETE https://[registry]:5000/v2/docker.io/library/busybox/manifests/sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101
HTTP/1.1 202 Accepted
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Thu, 13 Oct 2016 11:39:31 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
# docker run --rm registry:2 garbage-collect /etc/docker/registry/config.yml
time="2016-10-13T11:40:51Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/56/56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" go.version=go1.6.3 instance.id=f0bc6b6d-f7fa-4bb5-a63f-32f2e0ed8df0
time="2016-10-13T11:40:51Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/93/9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101" go.version=go1.6.3 instance.id=f0bc6b6d-f7fa-4bb5-a63f-32f2e0ed8df0
time="2016-10-13T11:40:51Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/e0/e02e811dd08fd49e7f6032625495118e63f597eb150403d02e3238af1df240ba" go.version=go1.6.3 instance.id=f0bc6b6d-f7fa-4bb5-a63f-32f2e0ed8df0
# curl -kXGET https://[registry]:5000/v2/docker.io/library/busybox/tags/list
{"name":"docker.io/library/busybox","tags":null}
# docker push [registry]:5000/docker.io/library/busybox
The push refers to a repository [[registry]:5000/docker.io/library/busybox]
e88b3f82283b: Layer already exists 
latest: digest: sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101 size: 505
# curl -kXGET https://[registry]:5000/v2/docker.io/library/busybox/tags/list
{"name":"docker.io/library/busybox","tags":["latest"]}
# curl -kXGET https://[registry]:5000/v2/docker.io/library/busybox/manifests/latest
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"docker.io/library/busybox","Revision":"sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101"}}]}
# docker restart local-registry
# docker push [registry]:5000/docker.io/library/busybox
The push refers to a repository [[registry]:5000/docker.io/library/busybox]
e88b3f82283b: Pushed 
latest: digest: sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101 size: 505

[wutongjie23hao]

The issue also exists.

[wutongjie23hao]

This issue cased by the registry cache. I disable the registry cache to solve this problem. And it has worked.

[milosgajdos]

Closing as wildly outdated. Please open a new issue after testing on the latest available release.