Docker-Content-Digest in GET does not match `docker push`

[7AC]

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.6
 Git commit:   20f81dd
 Built:        Sat Mar 12 04:07:44 UTC 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 20:39:58 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 6
 Running: 1
 Paused: 0
 Stopped: 5
Images: 6
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 50
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.17-boot2docker
Operating System: Boot2Docker 1.10.1 (TCL 6.4.1); master : b03e158 - Thu Feb 11 22:34:01 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.2 MiB
Name: dev
ID: OFRG:6OKA:ND64:QAPX:UPVB:QIZZ:OVNM:34VN:R33J:SVTU:XIW3:S6QI
Debug mode (server): true
 File Descriptors: 20
 Goroutines: 35
 System Time: 2016-03-25T01:12:13.364344828Z
 EventsListeners: 0
 Init SHA1: 
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Username: arastra
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox

Additional environment details (AWS, VirtualBox, physical, etc.):
VirtualBox 5.0.16 r105871 on Mac OS 10.11.4

Steps to reproduce the issue:

1. docker push 127.0.0.1:5000/foo/bar:12345
2. curl -D - http://127.0.0.1:5000/v2/foo/bar/manifests/12345 | grep Docker-Content-Digest
3. curl -X DELETE http://127.0.0.1:5000/v2/foo/bar/manifests/<digest from step 2>

Describe the results you received:
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}

Describe the results you expected:
Deletion of the image

Additional information you deem important (e.g. issue happens only occasionally):
The same sequence works by specifying the digest from the output docker push, which differs from the output of the GET request at step 2. I couldn't find any other digest that resembles the latter through the API.
I was redirected here by moby/moby#21496

[fishtek]

I'm running into this as well, as there doesn't seem to be a way to properly get the Digest via the API (w/o having captured the push information).

Looks the same as #1564

[fishtek]

Just figured this out... it appears you have to include an "accept" header for the newer version of the manifest schema... (probably if we have the newer client side that did the push), so that the repo calculates the proper digest.

I was able to get a Digest that matches my push (and which I can use for the Delete) like so:
curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -k -v -X GET https://dns-name-of-repo/v2/image-name/manifests/latest

[7AC]

Thanks @fishtek, that does work. However the response doesn't contain fsLayers hence no blobs. Does that mean that to have both the right digest and the blobs I need to issue two GETs with and without the Accept header?

[RichardScothern]

When you use curl to access the manifest, the registry is re-writing the manifest from version 2 to version 1 unless it sees that the client supports manifest schema 2 (supported by the Accept header. The digest returned is the digest of the content. When that header is provided, the manifest is returned as-is, and the digest is that of the actual manifest content stored by the registry.

@7AC the layers are listed within the layers array of the response.

[7AC]

Thanks, so basically I need to use layers and fallback to fsLayers? I also noticed that even after deleting all layers (except if mentioned by other images) and manifests the image still shows up in the _catalog. Is that expected? /v2/<name>/tags/list returns empty and /v2/<name>/manifests/<label> returns a 404.

[RichardScothern]

I'm not sure what you mean by fallback to fsLayers. fsLayers and layers are the same thing - the only difference is the format of the manifest. Also, a manifest can be pushed by digest, but have no tag information.

[7AC]

Ok, that answers my question: as a "deleter" I need to look for both: with the same Accept header I get the format with layers in some cases and fsLayers in some others. Is the image ever expected to disappear from the _catalog if for instance some of its layers stick around because are referenced by other images?

[7AC]

@RichardScothern thanks for the enhancement. Is deletion from the catalog and the tags list not supported though?

Giuseppes-Mac:test gvalente$ docker build -t registry.docker.sjc.aristanetworks.com:5000/foo/bar .                                                                                                                                            
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM alpine:latest
latest: Pulling from library/alpine
4d06f2521e4f: Already exists 
Digest: sha256:7739b19a213f3a0aa8dacbd5898c8bd467e6eaf71074296a3d75824e76257396
Status: Downloaded newer image for alpine:latest
 ---> 70c557e50ed6
Successfully built 70c557e50ed6
Giuseppes-Mac:test gvalente$ docker push registry.docker.sjc.aristanetworks.com:5000/foo/bar
The push refers to a repository [registry.docker.sjc.aristanetworks.com:5000/foo/bar]
8f045733649f: Mounted from test/test 
latest: digest: sha256:300f96719cd9297b942f67578f7e7fe0a4472f9c68c30aff78db728316279e6f size: 506
Giuseppes-Mac:test gvalente$ curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -k -v -X GET registry.docker.sjc.aristanetworks.com:5000/v2/foo/bar/manifests/latest
*   Trying 172.24.32.32...
* Connected to registry.docker.sjc.aristanetworks.com (172.24.32.32) port 5000 (#0)
> GET /v2/foo/bar/manifests/latest HTTP/1.1
> Host: registry.docker.sjc.aristanetworks.com:5000
> User-Agent: curl/7.43.0
> Accept: application/vnd.docker.distribution.manifest.v2+json
> 
< HTTP/1.1 200 OK
< Content-Length: 506
< Content-Type: application/vnd.docker.distribution.manifest.v2+json
< Docker-Content-Digest: sha256:300f96719cd9297b942f67578f7e7fe0a4472f9c68c30aff78db728316279e6f
< Docker-Distribution-Api-Version: registry/2.0
< Etag: "sha256:300f96719cd9297b942f67578f7e7fe0a4472f9c68c30aff78db728316279e6f"
< X-Content-Type-Options: nosniff
< Date: Wed, 30 Mar 2016 23:18:38 GMT
< 
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/octet-stream",
      "size": 1145,
      "digest": "sha256:70c557e50ed630deed07cbb0dc4d28aa0f2a485cf7af124cc48f06bce83f784b"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2319773,
         "digest": "sha256:4d06f2521e4f3ae47d0339ba887d234e11edfb8b3d9703ed856de95f4cc2d96d"
      }
   ]
* Connection #0 to host registry.docker.sjc.aristanetworks.com left intact
}Giuseppes-Mac:test gvalente$curl -X DELETE registry.docker.sjc.aristanetworks.com:5000/v2/foo/bar/blobs/sha256:4d06f2521e4f3ae47d0339ba887d234e11edfb8b3d9703ed856de95f4cc2d96d
Giuseppes-Mac:test gvalente$ curl -X DELETE registry.docker.sjc.aristanetworks.com:5000/v2/foo/bar/manifests/sha256:300f96719cd9297b942f67578f7e7fe0a4472f9c68c30aff78db728316279e6f
Giuseppes-Mac:test gvalente$ curl registry.docker.sjc.aristanetworks.com:5000/v2/_catalog
{"repositories":["foo/bar"]}
Giuseppes-Mac:test gvalente$ curl registry.docker.sjc.aristanetworks.com:5000/v2/foo/bar/tags/list
{"name":"foo/bar","tags":null}

[RichardScothern]

Currently you can completely gut a repository of it's manifests and blobs, but it will show up in the catalog since none of the surround structures are removed.

If a repository has no manifests, then it shouldn't appear in either (since there is nothing that can be pulled from the repository).

I'll file a bug to track this.

[ripatil]

@RichardScothern Do we have a fix for the above " Currently you can completely gut a repository of it's manifests and blobs, but it will show up in the catalog since none of the surround structures are removed." as i am also facing the same issue