private registry delete manifest results in broken state #1755

yofreke · 2016-05-31T00:27:12Z

Saw some similar issues, but didnt find anything for this particular problem.

docker registry version: registry:2 Digest: sha256:bf9b4a7b53a2f54c7b4d839103ca5be05b6a770ee0ba9c43e9ef23d602414f44

I am using a private registry backed by S3, using basic auth.

Steps:

list tags
do: curl https://<regsitry>/v2/manager/tags/list
observe: {"name":"manager","tags":[...,"1.3.0","1.3.1","latest"]}

I want to delete tag 1.3.1

get tag digest
do: curl https://<registry>/v2/manager/manifests/1.3.1 -v
observe: < Docker-Content-Digest: sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a
delete tag
do: curl -X DELETE https://<registry>/v2/manager/manifests/sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a -v
observe: < HTTP/1.1 202 Accepted
ensure tag is gone
do: curl https://<regsitry>/v2/manager/tags/list
observe: {"name":"manager","tags":[...,"1.3.0","1.3.1","latest"]}
server logs:

INFO[0907] response completed                            go.version=go1.6.2 http.request.host=<registry> http.request.id=63c217c1-b5ee-4d67-9a19-8fd79f210d87 http.request.method=GET http.request.remoteaddr=<my ip>:53486 http.request.uri=/v2/manager/tags/list http.request.useragent=curl/7.43.0 http.response.contenttype=application/json; charset=utf-8 http.response.duration=52.464518ms http.response.status=200 http.response.written=200 instance.id=6484f785-0dfd-4300-afde-13e8ad9da085 version=v2.4.1
67.161.4.204 - - [31/May/2016:00:24:31 +0000] "GET /v2/manager/tags/list HTTP/1.1" 200 200 "" "curl/7.43.0"

investigate why tag is still there
do: question life and the meaning of happiness, curl https://<registry>/v2/manager/manifests/1.3.1 -v
observe: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"manager","Revision":"sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a"}}]}
server logs:

ERRO[0940] response completed with error                 auth.user.name=<username> err.code=manifest unknown err.detail=unknown manifest name=manager revision=sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a err.message=manifest unknown go.version=go1.6.2 http.request.host=<registry> http.request.id=2a33440c-2d08-49d0-9ee0-61a9e6cf4996 http.request.method=GET http.request.remoteaddr=<my ip>:53492 http.request.uri=/v2/manager/manifests/1.3.1 http.request.useragent=curl/7.43.0 http.response.contenttype=application/json; charset=utf-8 http.response.duration=30.85648ms http.response.status=404 http.response.written=183 instance.id=6484f785-0dfd-4300-afde-13e8ad9da085 vars.name=manager vars.reference=1.3.1 version=v2.4.1
67.161.4.204 - - [31/May/2016:00:25:04 +0000] "GET /v2/manager/manifests/1.3.1 HTTP/1.1" 404 183 "" "curl/7.43.0"

Questions:
How do I actually remove this tag?

The text was updated successfully, but these errors were encountered:

RichardScothern · 2016-05-31T18:27:46Z

hi @yofreke this is not the expected behavior. When a manifest is removed, any tags which reference it are also removed.

If you have a minimal reproduction case could paste the file layout of your registry?

yofreke · 2016-05-31T20:35:25Z

The registry we are using has >100gb worth of data. I can do some investigating if there are specifics you are looking for?

oxygen0211 · 2016-06-01T07:09:05Z

+1

Seeing the exact same problem with registry:2.4.1 (ID b0bfd772479f4e19f5546daf8c2a0c9673e699006de30d1f5b1623ee6d8d27f8) with a fairly big amount of images (~550GB).

Probably will have to ignore this error for now in my cleanup script so that our registry won't run out of HDD

RichardScothern · 2016-06-01T16:58:51Z

@yofreke : i'm curious to see the repository layout, specifically the tags and manifests.

Otherwise, if you can post the registry logs with the most verbose setting that would help too.

Grief · 2016-06-06T13:29:41Z

I am observing the same issue, and I believe that the same steps lead me to it. But I cannot find any clue except that directories for tags still contain link files with sha256 which were removed. I'd bet that curl -X DELETE https://<registry>/v2/manager/manifests/sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a -v didn't delete tags, but deleted manifest only.

RichardScothern · 2016-06-06T17:17:35Z

The delete command removes the link file of the manifest and then the tags which are referencing that manifest. If you inspect the storage backend for the repository in question and can still see tags contain the manifest then there is a bug.

If you do see this, please post your registry version, logs from the delete operation and if possible the repository keys in the back-end. Without this information, debugging will be very difficult.

kuijp · 2016-07-13T08:50:24Z

+1

Having the same issue with same commands. Registry version 2.4.1

sen-lgtm · 2016-07-13T11:33:03Z

FYI
This command

 curl -v https://<registry>/v2/<repo>/manifests/<tag>

does not provide the right tag digest for v2 deletion. You have to use a additional header as mentionned in the offical docs: https://docs.docker.com/registry/spec/api/#/deleting-an-image

curl -v https://<registry>/v2/<repo>/manifests/<tag> -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'

Proceeding this way fixed the issue for me.

jessegit · 2017-01-04T12:31:54Z

This was happening to me, too. Initially tried to delete with registry:2.4.1 and some of the tags were stuck. Upgrade to registry:2.5.1 seems to have fixed it for me.

stevvooe · 2017-01-10T20:02:22Z

Sounds like this is fixed in 2.5.1. Closing.

If anyone is still having issues, please let me know.

jostyee · 2017-05-06T06:52:45Z

We're experiencing this bug in Registry v2.6.1.

curl -X GET "localhost:5000/v2/thirdparty/gitlab/tags/list"
{"name":"thirdparty/gitlab","tags":["8.17","9.0.5","9.1.1","9.1.2"]}

returns 4 tags, when fetching manifests of tag 8.17 as #1755 (comment) said, it failed:

curl -v "localhost:5000/v2/thirdparty/gitlab/manifests/8.17" -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> GET /v2/thirdparty/gitlab/manifests/8.17 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 404 Not Found
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Date: Sat, 06 May 2017 06:49:36 GMT
< Content-Length: 193
<
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"thirdparty/gitlab","Revision":"sha256:1f2291aa0094f379a331772b3e34dfd68d26594055388d74877d5d05b2de72c0"}}]}
* Connection #0 to host localhost left intact

docker logs of registry as below:

time="2017-05-06T06:54:53Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=thirdparty/gitlab revision=sha256:1f2291aa0094f379a331772b3e34dfd68d26594055388d74877d5d05b2de72c0" err.message="manifest unknown" go.version=go1.7.3 http.request.host="localhost:5000" http.request.id=dff260f1-b89a-450a-bad5-7446fd7f7283 http.request.method=GET http.request.remoteaddr="172.18.0.1:50522" http.request.uri="/v2/thirdparty/gitlab/manifests/8.17" http.request.useragent="curl/7.35.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=45.967892ms http.response.status=404 http.response.written=193 instance.id=c02efd08-a6e7-4ac0-86ce-75427ca2d6bb vars.name="thirdparty/gitlab" vars.reference=8.17 version=v2.6.1
172.18.0.1 - - [06/May/2017:06:54:53 +0000] "GET /v2/thirdparty/gitlab/manifests/8.17 HTTP/1.1" 404 193 "" "curl/7.35.0"

How can I delete this tag now?

RainerW · 2017-09-08T14:12:37Z

Got the same problem in v.2.6.2. Running the garbage collector will also not free diskspace ... not good!
Removing the Manifest Blob folder will also free disk, but this requires FS access which means the script knows how the registry is storing data!

Kanshiroron · 2017-10-03T09:41:45Z

@stevvooe
+1 with v2.6.2

stevvooe · 2017-10-03T17:02:44Z

@hinshun Could you please take a peak at this issue?

hinshun · 2017-10-04T21:27:18Z

Hi all, thanks for reporting the issue. Unfortunately I was unable to reproduce the issue.

I made a minimal config.yml where I deployed registry with:
docker run --rm -it -v $(pwd)/config.yml:/etc/docker/registry/config.yml -p 5000:5000 --name registry registry:2.6.2

Configuration:

version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  s3:
    region: <stripped>
    bucket: <stripped>
    accesskey: <stripped>
    secretkey: <stripped>
http:
  addr: :5000

My steps:

$ docker pull alpine:3.5
$ docker pull alpine:3.6
$ docker tag alpine 3.5 localhost:5000/some/repo:1
$ docker tag alpine 3.6 localhost:5000/some/repo:2
$ docker push localhost:5000/some/repo:1
$ docker push localhost:5000/some/repo:2

$ curl localhost:5000/v2/some/repo/tags/list
{"name":"some/repo","tags":["1","2"]}
$ curl -v "localhost:5000/v2/some/repo/manifests/1" -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5000 (#0)
> GET /v2/some/repo/manifests/1 HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.54.1
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 200 OK
< Content-Length: 528
< Content-Type: application/vnd.docker.distribution.manifest.v2+json
< Docker-Content-Digest: sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b
< Docker-Distribution-Api-Version: registry/2.0
< Etag: "sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b"
< Date: Wed, 04 Oct 2017 21:22:20 GMT
<
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1511,
      "digest": "sha256:37c7be7a096b71f4e12e957c5ac83c3a7b01d8ae9f43a68c319e298d61b608fe"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1970271,
         "digest": "sha256:ab14e39f58e6d8ba465d2bb577a82a750ec0bcd2342b380920f9e7f307be3c4f"
      }
   ]
* Connection #0 to host localhost left intact
}

$ curl -X DELETE -v localhost:5000/v2/some/repo/manifests/sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5000 (#0)
> DELETE /v2/some/repo/manifests/sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.54.1
> Accept: */*
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0
< Date: Wed, 04 Oct 2017 21:23:15 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact 

$ curl localhost:5000/v2/some/repo/tags/list
{"name":"some/repo","tags":["2"]}

If you are still experiencing problems, please open a new issue and provide your configuration and steps to reproduce the issue. It's likely there are differences between my minimal setup and specific configurations that triggering the bug.

Davidrjx · 2018-01-23T02:53:22Z

i also use registry 2.6.2 and following @hinshun 's solution, it works for me.
however, i am not clear whether an image is fully deleted from disk and what the cleanup operation contains via curl -X DELETE https_server:port/v2/<name>/manifests/digest.
so i think both manifests and blobs are erased ,that indicates an image from local private registry is deleted as docker image rm , right?
still please some experts provide some ideas,thanks

imendi · 2018-08-23T19:55:09Z

Same thing happens to me

registry version -> registry github.com/docker/distribution v2.6.2

docker version -> Docker version 18.03.1-ce, build 9ee9f40

using -> -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'

ooxiaohan · 2019-01-16T05:35:26Z

Same thing happens to me!

manishtomar · 2019-01-16T07:28:25Z

@hinshun @imendi @ooxiaohan The behavior reported by @hinshun is expected. When a manifest is deleted then all the tags that refer to it are also deleted. In his/her case the manifest referred to tag 1 and hence it was deleted.

@Davidrjx When you do curl -X DELETE https_server:port/v2/<name>/manifests/digest the reference to manifest and tag are deleted. However, their content still remains in blobs/ directory. docker image rm is independent of what is in registry and is about how docker engine manages images on your dev machine.

Oleksii-Terekhov · 2019-01-18T10:23:25Z

@manishtomar problem with GET .../list/... - I have same thing.
LIST->GET->DELETE->stop->gc->start
Disk free, but some tags in list, .../manifests/deleted_tag -> 404 with MANIFEST_UNKNOWN

a1kemist · 2020-04-15T03:41:13Z

I can confirm that we are also seeing the same issue with our private registry (version 2.7.1) and using the proper headers, etc.

Note that this doesn't happen on every image deletion. Most deletes are successful and remove the manifest/tag together but some get "stuck" in this state. One workaround solution that we have found is to push another "dummy" image to the "stuck" tag and then retry the deletion. This has worked so far, but I'm unsure if the blobs from the old tag are actually deleted on the storage or have become orphaned by this "workaround".

tkakantousis · 2020-08-13T13:19:23Z

@a1kemist
Having the same issue (version 2.7.1). Removing images is inconsistent, in that garbage collection (even with the -m flag) usually does not remove any blobs/manifests and when pushing and pulling the image again we get manifest unknown error.

jcarsique · 2021-02-26T17:53:31Z

Please tell me if I should open a new issue "Tag manifest unknown after deletion, and growing disk usage".

I have a similar situation on version 2.6.2, some deleted tags are still listed but they raise manifest unknown.
Typically this happens on disposable tags created Skaffold caching.
/var/lib/registry is consuming too much disk space, the garbage collector does not seem to be able to resolve this issue and do its cleanup. Only a few tags are kept but the disk usage is going huge.

I redacted the registry and repository names:

curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' https://REGISTRY/v2/REPO/tags/list
{"name":"REPO","tags":["983021bb257fe7fb5446ceec64894b14eb65ccd2e0025885415cdb5c2876c534", ...]}

curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' https://REGISTRY/v2/REPO/manifests/983021bb257fe7fb5446ceec64894b14eb65ccd2e0025885415cdb5c2876c534
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"REPO","Revision":"sha256:eb6f5361739bde88b438dd7f2abf6dd9989540e78cb68216cd65b2d3e27bfd2f"}}]}

On the server, some files remain in /var/lib/registry/docker/registry/v2/repositories/ (_manifests and _layers).
I tried:

rm -rf /var/lib/registry/docker/registry/v2/repositories/REPO/_manifests/tags/TAG/index/sha256/SHA
rm -r /var/lib/registry/docker/registry/v2/repositories/REPO/_manifests/revisions/sha256/SHA
/bin/registry garbage-collect /etc/docker/registry/config.yml

What can be done on the filesystem to remove some tags without corrupting the filesystem?
How to manually on the filesystem

remove a tag?
remove a repository?
garbage unused layers and blobs ? Isn't this the purpose of the garbage collector?

joaodrp · 2021-02-26T18:08:55Z

@jcarsique, what's the result of those requests if you omit the Accept header?

sudo-bmitch · 2021-02-26T23:28:09Z

@jcarsique I see that when I accidentally deleted a single platform manifest from a multi platform manifest list. You can pull the digest for the manifest list with the proper accept header, and then delete that digest to clean things up.

jcarsique · 2021-03-01T10:23:41Z

@joaodrp same result without headers:

$ curl -s https://REGISTRY/v2/REPO/tags/list
{"name":"REPO","tags":["7451eb889ccf9539459ccd62aba281beb68b1c98b34d3128bd7d105deb232dc6"]}
$ curl -s https://REGISTRY/v2/REPO/manifests/7451eb889ccf9539459ccd62aba281beb68b1c98b34d3128bd7d105deb232dc6
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"REPO","Revision":"sha256:9ef53bd1a6a6e2c8827250315d52cac6fd9a5623831984caa08e3910e66c66ef"}}]}

jcarsique · 2021-03-01T10:34:06Z

@sudo-bmitch what do you mean? My issue is that no digest is returned but manifest unknown, with this accept header: 'Accept: application/vnd.docker.distribution.manifest.v2+json'

sudo-bmitch · 2021-03-01T11:46:10Z

@jcarsique the header for a docker manifest list is Accept: application/vnd.docker.distribution.manifest.list.v2+json. Here's an example cleaning v0.2.0 from this local repo:

$ curl -s http://localhost:5000/v2/regclient/regsync/tags/list
{"name":"regclient/regsync","tags":["v0.2.0","copy","latest"]} 

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
{       
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",       
   "schemaVersion": 2, 
   "config": {    
      "mediaType": "application/vnd.docker.container.image.v1+json",        
      "digest": "sha256:55dff1bf550e642373247f3d30ae809d2ba353dfaba89c85e2496e1cbf1e4550",
      "size": 3025                  
   },                                                               
   "layers": [                                                                       
...

$ curl -s http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0                                                                                                                                                                                                           
{                                                                                                                                                                                                                                                                               
   "schemaVersion": 1,                                                                                                                                                                                                                                                          
   "name": "regclient/regsync",                                                                                                                                                                                                                                                 
   "tag": "v0.2.0",                                                                                                                                                                                                                                                             
   "architecture": "amd64",                                                                                                                                                                                                                                                     
   "fsLayers": [                                                                                                                                                                                                                                                                
...

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0                                                                                                                                    
{                                                                                                                                                                                                                                                                               
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",                                                                                                                                                                                                    
   "schemaVersion": 2,                                                                                                                                                                                                                                                          
   "manifests": [                                                                                                                                                                                                                                                               
      {                                                                                                                                                                                                                                                                         
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                                                                                                                                                                                                   
         "digest": "sha256:615f1aa6e32c5596668a30b31f123e5920eb15ffff78710722f2c6afa8a8d8ee",                                                                                                                                                                                   
         "size": 1152,                                                                                                                                                                                                                                                          
         "platform": {                                                                                                                                                                                                                                                          
            "architecture": "386",                                                                                                                                                                                                                                              
            "os": "linux"                                                                                                                                                                                                                                                       
         }                                                                                                                                                                                                                                                                      
      },                                                             
      {                                                                
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                 
         "digest": "sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e",                                                                                                                                                                                   
         "size": 1152,                                                         
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
...

Same image, three different manifests depending on the header, and the manifest list is the real one, the registry is looking up the default platform for the v2 manifest, and automatically converting that to a v1 schema without any header. Those manifests have different digests:

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 2342
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Docker-Content-Digest: sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:34:48 GMT

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 1152
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:34:55 GMT

Deleting the digest for a single platform leaves things in a semi-broken state:

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 1152
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:34:55 GMT

$ curl -s -X DELETE http://localhost:5000/v2/regclient/regsync/manifests/sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 404 Not Found
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:36:24 GMT
Content-Length: 193

$ curl -s http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"regclient/regsync","Revision":"sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e"}}]}

$ curl -s http://localhost:5000/v2/regclient/regsync/tags/list
{"name":"regclient/regsync","tags":["v0.2.0","copy","latest"]}

But you can still get the manifest list digest and remove that to clean things up:

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 2342
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Docker-Content-Digest: sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:36:41 GMT

$ curl -s -X DELETE http://localhost:5000/v2/regclient/regsync/manifests/sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b

$ curl -s  http://localhost:5000/v2/regclient/regsync/tags/list
{"name":"regclient/regsync","tags":["copy","latest"]}

jcarsique · 2021-03-01T13:27:01Z

@sudo-bmitch It's crystal clear, thank you. I'll take care to use the proper digest for deletion, when using the API (I'm now mostly using the reg tool).
However in the current case, the manifest list digest is not either returned.
On the filesystem, I can find this:

$ find . -name 9ef53bd1a6a6e2c8827250315d52cac6fd9a5623831984caa08e3910e66c66ef
REPO/_manifests/revisions/sha256/9ef53bd1a6a6e2c8827250315d52cac6fd9a5623831984caa08e3910e66c66ef
REPO/_manifests/tags/7451eb889ccf9539459ccd62aba281beb68b1c98b34d3128bd7d105deb232dc6/index/sha256/9ef53bd1a6a6e2c8827250315d52cac6fd9a5623831984caa08e3910e66c66ef

Should I simply remove those two directories?

I don't know if it's related or another issue but there's also a lot of revisions not referenced by a tag.
Would it be correct and enough for garbage collecting to remove those like this ?

for f in $(find . -path "*/_manifests/revisions/sha256/*" -type f); do f2=$(cat $f); [ ! -d */_manifests/tags/*/*/sha256/${f2#sha256:} ] && dirname $f; done | xargs rm -rf

sudo-bmitch · 2021-03-01T13:46:36Z

@jcarsique There's also the OCI media types. It's a bit self promoting, but regclient's regctl is what I use for this, lot of similar functionality to reg. There, the --list flag is needed when running image digest to get the correct digest to delete.

Double check that the digest you deleted matches what you're seeing in those repos, e.g.

curl -s -X DELETE http://registry:5000/v2/REPO/manifests/sha256:9ef53bd1a6a6e2c8827250315d52cac6fd9a5623831984caa08e3910e66c66ef

Once the manifest is removed for me, the tag is gone immediately without any GC. The GC is needed to remove some of the underlying blobs. I don't know enough of the filesystem storage of registry to be able to advise what's safe to remove.

jcarsique · 2021-03-02T13:18:45Z

@sudo-bmitch by the way, regctl succeeded to delete anything properly 👏 : I will look at regbot that sounds great for automation. Thank you.

I also found this cleanup.sh script that works on the local filesystem. It tends to give information on the file structure.

At the end, upgrading from 2.6 to 2.7 allowed to use the new /bin/registry garbage-collect -m option that freed up 75% of the storage!

stevvooe closed this as completed Jan 10, 2017

mmpei mentioned this issue Dec 11, 2018

Delete tag failed -- size 0 image. goharbor/harbor#6515

Closed

nisc-acooper mentioned this issue May 31, 2019

The size are 0 images always in the registry. goharbor/harbor#7924

Closed

private registry delete manifest results in broken state #1755

private registry delete manifest results in broken state #1755

yofreke commented May 31, 2016

RichardScothern commented May 31, 2016

yofreke commented May 31, 2016

oxygen0211 commented Jun 1, 2016

RichardScothern commented Jun 1, 2016

Grief commented Jun 6, 2016

RichardScothern commented Jun 6, 2016

kuijp commented Jul 13, 2016

sen-lgtm commented Jul 13, 2016

jessegit commented Jan 4, 2017

stevvooe commented Jan 10, 2017

jostyee commented May 6, 2017 •

edited

RainerW commented Sep 8, 2017 •

edited

Kanshiroron commented Oct 3, 2017 •

edited

stevvooe commented Oct 3, 2017

hinshun commented Oct 4, 2017

Davidrjx commented Jan 23, 2018 •

edited

imendi commented Aug 23, 2018

ooxiaohan commented Jan 16, 2019

manishtomar commented Jan 16, 2019

Oleksii-Terekhov commented Jan 18, 2019

a1kemist commented Apr 15, 2020

tkakantousis commented Aug 13, 2020

jcarsique commented Feb 26, 2021

joaodrp commented Feb 26, 2021

sudo-bmitch commented Feb 26, 2021

jcarsique commented Mar 1, 2021

jcarsique commented Mar 1, 2021 •

edited

sudo-bmitch commented Mar 1, 2021

jcarsique commented Mar 1, 2021 •

edited

sudo-bmitch commented Mar 1, 2021

jcarsique commented Mar 2, 2021

private registry delete manifest results in broken state #1755

private registry delete manifest results in broken state #1755

Comments

yofreke commented May 31, 2016

RichardScothern commented May 31, 2016

yofreke commented May 31, 2016

oxygen0211 commented Jun 1, 2016

RichardScothern commented Jun 1, 2016

Grief commented Jun 6, 2016

RichardScothern commented Jun 6, 2016

kuijp commented Jul 13, 2016

sen-lgtm commented Jul 13, 2016

jessegit commented Jan 4, 2017

stevvooe commented Jan 10, 2017

jostyee commented May 6, 2017 • edited

RainerW commented Sep 8, 2017 • edited

Kanshiroron commented Oct 3, 2017 • edited

stevvooe commented Oct 3, 2017

hinshun commented Oct 4, 2017

Davidrjx commented Jan 23, 2018 • edited

imendi commented Aug 23, 2018

ooxiaohan commented Jan 16, 2019

manishtomar commented Jan 16, 2019

Oleksii-Terekhov commented Jan 18, 2019

a1kemist commented Apr 15, 2020

tkakantousis commented Aug 13, 2020

jcarsique commented Feb 26, 2021

joaodrp commented Feb 26, 2021

sudo-bmitch commented Feb 26, 2021

jcarsique commented Mar 1, 2021

jcarsique commented Mar 1, 2021 • edited

sudo-bmitch commented Mar 1, 2021

jcarsique commented Mar 1, 2021 • edited

sudo-bmitch commented Mar 1, 2021

jcarsique commented Mar 2, 2021

jostyee commented May 6, 2017 •

edited

RainerW commented Sep 8, 2017 •

edited

Kanshiroron commented Oct 3, 2017 •

edited

Davidrjx commented Jan 23, 2018 •

edited

jcarsique commented Mar 1, 2021 •

edited

jcarsique commented Mar 1, 2021 •

edited