Internal Server Error when getting blobs as manifests

[brackendawson]

When you get a manifest by digest, but use the digest of an image config or layer instead of the manifest, you get a 500 error.

Version info

alandaws@alansmbp ~/test-docker/smol $ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
alandaws@alansmbp ~/test-docker/smol $ docker info
Client:
 Debug Mode: false

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 135
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.184-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 1.952GiB
 Name: docker-desktop
 ID: CZCL:7WGU:YTDN:FRAJ:4UVB:QN3A:CF7E:KXTW:SG2V:ZHGM:MGCN:4J3V
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 36
  Goroutines: 50
  System Time: 2020-03-11T10:31:35.9688813Z
  EventsListeners: 2
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

alandaws@alansmbp ~/test-docker/smol $ docker exec -it d70 registry --version
registry github.com/docker/distribution v2.7.1

Steps to reproduce

1. Build an image and push it to a v2 registry.
2. Find the digest of the image config and pull it like it was a manifest: curl http://<registry>/v2/<repo>/manifests/sha256:<digest_of_config>
3. Find the digest of one of the layers (it might need to be a small layer) and repeat the above.

What happened

I get a 500 error with the digest of layers and the image config.

If I make up a digest I get 404, if I use the manifest's digest I get 200 and the manifest.

Console output

alandaws@alansmbp ~/test-docker/smol $ cat Dockerfile 
FROM scratch

COPY smol .
alandaws@alansmbp ~/test-docker/smol $ docker build -t localhost:5000/smol .
Sending build context to Docker daemon   2.56kB
Step 1/2 : FROM scratch
 ---> 
Step 2/2 : COPY smol .
 ---> Using cache
 ---> 9cb6208bf9db
Successfully built 9cb6208bf9db
Successfully tagged localhost:5000/smol:latest
alandaws@alansmbp ~/test-docker/smol $ ###################### Start a registry
alandaws@alansmbp ~/test-docker/smol $ docker run -dp 5000:5000 registry:2.7.1
Unable to find image 'registry:2.7.1' locally
2.7.1: Pulling from library/registry
Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
Status: Downloaded newer image for registry:2.7.1
d70afdb701d6f5a65b6a97ac2370b6e9b592740d30c7528d5d61f3d454ad0a64
alandaws@alansmbp ~/test-docker/smol $ ###################### Push the image
alandaws@alansmbp ~/test-docker/smol $ docker push localhost:5000/smol
The push refers to repository [localhost:5000/smol]
4ffbc913c9e6: Pushed 
latest: digest: sha256:082be0cc7af031e60741038131ea51218ac261906c5b1d19dd09fbf9d87a99f4 size: 523
alandaws@alansmbp ~/test-docker/smol $ ######################  Can curl the manifest OK
alandaws@alansmbp ~/test-docker/smol $ curl -D- http://localhost:5000/v2/smol/manifests/sha256:082be0cc7af031e60741038131ea51218ac261906c5b1d19dd09fbf9d87a99f4
HTTP/1.1 200 OK
Content-Length: 523
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:082be0cc7af031e60741038131ea51218ac261906c5b1d19dd09fbf9d87a99f4
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:082be0cc7af031e60741038131ea51218ac261906c5b1d19dd09fbf9d87a99f4"
X-Content-Type-Options: nosniff
Date: Wed, 11 Mar 2020 10:29:47 GMT

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1178,
      "digest": "sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 98,
         "digest": "sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d"
      }
   ]
}
alandaws@alansmbp ~/test-docker/smol $ ###################### Curling the config sha is a server error
alandaws@alansmbp ~/test-docker/smol $ curl -D- http://localhost:5000/v2/smol/manifests/sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf
HTTP/1.1 500 Internal Server Error
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 11 Mar 2020 10:30:00 GMT
Content-Length: 244

{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/smol/_manifests/tags/ssha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf/current/link","DriverName":"filesystem"}}]}
alandaws@alansmbp ~/test-docker/smol $ ###################### Curling the layer sha is a server error
alandaws@alansmbp ~/test-docker/smol $ curl -D- http://localhost:5000/v2/smol/manifests/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d
HTTP/1.1 500 Internal Server Error
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 11 Mar 2020 10:30:15 GMT
Content-Length: 80

{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Offset":1}}]}
alandaws@alansmbp ~/test-docker/smol $ ###################### Curling a made up sha is OK
alandaws@alansmbp ~/test-docker/smol $ curl -D- http://localhost:5000/v2/smol/manifests/sha256:4ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771e
HTTP/1.1 404 Not Found
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 11 Mar 2020 10:30:30 GMT
Content-Length: 180

{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"smol","Revision":"sha256:4ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771e"}}]}

Registry logs

time="2020-03-11T10:29:11.7142784Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.11.2 instance.id=f43b2bc3-0ba0-440c-9d1c-d3178efdfa9e service=registry version=v2.7.1 
time="2020-03-11T10:29:11.7144443Z" level=info msg="redis not configured" go.version=go1.11.2 instance.id=f43b2bc3-0ba0-440c-9d1c-d3178efdfa9e service=registry version=v2.7.1 
time="2020-03-11T10:29:11.7160262Z" level=info msg="Starting upload purge in 46m0s" go.version=go1.11.2 instance.id=f43b2bc3-0ba0-440c-9d1c-d3178efdfa9e service=registry version=v2.7.1 
time="2020-03-11T10:29:11.7224809Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.11.2 instance.id=f43b2bc3-0ba0-440c-9d1c-d3178efdfa9e service=registry version=v2.7.1 
time="2020-03-11T10:29:11.7228705Z" level=info msg="listening on [::]:5000" go.version=go1.11.2 instance.id=f43b2bc3-0ba0-440c-9d1c-d3178efdfa9e service=registry version=v2.7.1 
time="2020-03-11T10:29:25.9322976Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=42d63f3c-7c11-4f3e-9a55-38f81d0075c1 http.request.method=GET http.request.remoteaddr="172.17.0.1:60980" http.request.uri="/v2/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.1562ms http.response.status=200 http.response.written=2 
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "HEAD /v2/smol/blobs/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d HTTP/1.1" 404 157 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:25.9432664Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=8ff8e117-b566-4802-b402-6cd348a43a15 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:60984" http.request.uri="/v2/smol/blobs/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.0927ms http.response.status=404 http.response.written=157 vars.digest="sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d" vars.name=smol 
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "POST /v2/smol/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:25.9548709Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=c3778457-cde6-4f47-83f5-5f17f2426722 http.request.method=POST http.request.remoteaddr="172.17.0.1:60988" http.request.uri="/v2/smol/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=7.8684ms http.response.status=202 http.response.written=0 
time="2020-03-11T10:29:25.9676915Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=2ea238b6-b9f3-4e29-8fd8-94361ab33c8d http.request.method=PATCH http.request.remoteaddr="172.17.0.1:60992" http.request.uri="/v2/smol/blobs/uploads/daa12b37-298c-4e23-99e4-cc00626fe735?_state=oiWt1daIgyBfxlnNyn823Zr8mDmMt00UFRUYc_CzbnZ7Ik5hbWUiOiJzbW9sIiwiVVVJRCI6ImRhYTEyYjM3LTI5OGMtNGUyMy05OWU0LWNjMDA2MjZmZTczNSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wMy0xMVQxMDoyOToyNS45NDk0MTlaIn0%3D" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=7.8698ms http.response.status=202 http.response.written=0 
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "PATCH /v2/smol/blobs/uploads/daa12b37-298c-4e23-99e4-cc00626fe735?_state=oiWt1daIgyBfxlnNyn823Zr8mDmMt00UFRUYc_CzbnZ7Ik5hbWUiOiJzbW9sIiwiVVVJRCI6ImRhYTEyYjM3LTI5OGMtNGUyMy05OWU0LWNjMDA2MjZmZTczNSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wMy0xMVQxMDoyOToyNS45NDk0MTlaIn0%3D HTTP/1.1" 202 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:25.9807881Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=8aee2a64-9051-4b9e-8aea-12902e1f89c5 http.request.method=PUT http.request.remoteaddr="172.17.0.1:60996" http.request.uri="/v2/smol/blobs/uploads/daa12b37-298c-4e23-99e4-cc00626fe735?_state=E3b4d7rJOpDnVE_w8cm41vaYpdJ3A4F7dQ2PKKT1JIV7Ik5hbWUiOiJzbW9sIiwiVVVJRCI6ImRhYTEyYjM3LTI5OGMtNGUyMy05OWU0LWNjMDA2MjZmZTczNSIsIk9mZnNldCI6OTgsIlN0YXJ0ZWRBdCI6IjIwMjAtMDMtMTFUMTA6Mjk6MjVaIn0%3D&digest=sha256%3A3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=9.3818ms http.response.status=201 http.response.written=0 
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "PUT /v2/smol/blobs/uploads/daa12b37-298c-4e23-99e4-cc00626fe735?_state=E3b4d7rJOpDnVE_w8cm41vaYpdJ3A4F7dQ2PKKT1JIV7Ik5hbWUiOiJzbW9sIiwiVVVJRCI6ImRhYTEyYjM3LTI5OGMtNGUyMy05OWU0LWNjMDA2MjZmZTczNSIsIk9mZnNldCI6OTgsIlN0YXJ0ZWRBdCI6IjIwMjAtMDMtMTFUMTA6Mjk6MjVaIn0%3D&digest=sha256%3A3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d HTTP/1.1" 201 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:25.984394Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=fa56a488-6d10-4299-8b8c-79d4555e9d84 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:32768" http.request.uri="/v2/smol/blobs/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.contenttype="application/octet-stream" http.response.duration=1.1256ms http.response.status=200 http.response.written=0 
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "HEAD /v2/smol/blobs/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d HTTP/1.1" 200 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:25.9950254Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=3e3488cc-9f9a-44ac-8795-a5665f6d2cde http.request.method=HEAD http.request.remoteaddr="172.17.0.1:32772" http.request.uri="/v2/smol/blobs/sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.5316ms http.response.status=404 http.response.written=157 vars.digest="sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf" vars.name=smol 
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "HEAD /v2/smol/blobs/sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf HTTP/1.1" 404 157 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
172.17.0.1 - - [11/Mar/2020:10:29:25 +0000] "POST /v2/smol/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:26.002762Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=3878f1f1-701a-4a92-b8e5-fc2f1a33d682 http.request.method=POST http.request.remoteaddr="172.17.0.1:32776" http.request.uri="/v2/smol/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=4.6251ms http.response.status=202 http.response.written=0 
time="2020-03-11T10:29:26.0093297Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=cdbec257-1a9e-483e-852a-05967ab97b76 http.request.method=PATCH http.request.remoteaddr="172.17.0.1:32780" http.request.uri="/v2/smol/blobs/uploads/1fb890b0-e5a4-49d3-8704-281f606b7cca?_state=q2tp1yfumk-N6d6iQ1G2qwwodU4CrEe_9OZ_JYLEzV17Ik5hbWUiOiJzbW9sIiwiVVVJRCI6IjFmYjg5MGIwLWU1YTQtNDlkMy04NzA0LTI4MWY2MDZiN2NjYSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wMy0xMVQxMDoyOToyNS45OTg3NDU5WiJ9" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=4.2208ms http.response.status=202 http.response.written=0 
172.17.0.1 - - [11/Mar/2020:10:29:26 +0000] "PATCH /v2/smol/blobs/uploads/1fb890b0-e5a4-49d3-8704-281f606b7cca?_state=q2tp1yfumk-N6d6iQ1G2qwwodU4CrEe_9OZ_JYLEzV17Ik5hbWUiOiJzbW9sIiwiVVVJRCI6IjFmYjg5MGIwLWU1YTQtNDlkMy04NzA0LTI4MWY2MDZiN2NjYSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wMy0xMVQxMDoyOToyNS45OTg3NDU5WiJ9 HTTP/1.1" 202 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:26.016925Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e2e7ae00-e2c0-4fb1-9c8c-82c30d802d91 http.request.method=PUT http.request.remoteaddr="172.17.0.1:32784" http.request.uri="/v2/smol/blobs/uploads/1fb890b0-e5a4-49d3-8704-281f606b7cca?_state=Rfuzk6M5n71DfCCH645b38JxybaloMiY89c4YRSEihd7Ik5hbWUiOiJzbW9sIiwiVVVJRCI6IjFmYjg5MGIwLWU1YTQtNDlkMy04NzA0LTI4MWY2MDZiN2NjYSIsIk9mZnNldCI6MTE3OCwiU3RhcnRlZEF0IjoiMjAyMC0wMy0xMVQxMDoyOToyNVoifQ%3D%3D&digest=sha256%3A9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=5.024ms http.response.status=201 http.response.written=0 
172.17.0.1 - - [11/Mar/2020:10:29:26 +0000] "PUT /v2/smol/blobs/uploads/1fb890b0-e5a4-49d3-8704-281f606b7cca?_state=Rfuzk6M5n71DfCCH645b38JxybaloMiY89c4YRSEihd7Ik5hbWUiOiJzbW9sIiwiVVVJRCI6IjFmYjg5MGIwLWU1YTQtNDlkMy04NzA0LTI4MWY2MDZiN2NjYSIsIk9mZnNldCI6MTE3OCwiU3RhcnRlZEF0IjoiMjAyMC0wMy0xMVQxMDoyOToyNVoifQ%3D%3D&digest=sha256%3A9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf HTTP/1.1" 201 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
172.17.0.1 - - [11/Mar/2020:10:29:26 +0000] "HEAD /v2/smol/blobs/sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf HTTP/1.1" 200 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2020-03-11T10:29:26.0216712Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=23400062-960f-4b37-bea3-7124a90a3f88 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:32788" http.request.uri="/v2/smol/blobs/sha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.contenttype="application/octet-stream" http.response.duration=2.5402ms http.response.status=200 http.response.written=0 
time="2020-03-11T10:29:26.0322406Z" level=info msg="response completed" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="localhost:5000" http.request.id=322a0639-09e4-4ee3-a771-0ab2d4b3d00d http.request.method=PUT http.request.remoteaddr="172.17.0.1:32792" http.request.uri="/v2/smol/manifests/latest" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=8.3615ms http.response.status=201 http.response.written=0 
172.17.0.1 - - [11/Mar/2020:10:29:26 +0000] "PUT /v2/smol/manifests/latest HTTP/1.1" 201 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
172.17.0.1 - - [11/Mar/2020:10:29:47 +0000] "GET /v2/smol/manifests/sha256:082be0cc7af031e60741038131ea51218ac261906c5b1d19dd09fbf9d87a99f4 HTTP/1.1" 200 523 "" "curl/7.64.1"
time="2020-03-11T10:29:47.4219866Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=a1637c72-1013-4bdb-8378-01c01bd44e13 http.request.method=GET http.request.remoteaddr="172.17.0.1:32794" http.request.uri="/v2/smol/manifests/sha256:082be0cc7af031e60741038131ea51218ac261906c5b1d19dd09fbf9d87a99f4" http.request.useragent="curl/7.64.1" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=1.4826ms http.response.status=200 http.response.written=523 
172.17.0.1 - - [11/Mar/2020:10:30:00 +0000] "GET /v2/smol/manifests/ssha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf HTTP/1.1" 500 244 "" "curl/7.64.1"
time="2020-03-11T10:30:00.1049556Z" level=error msg="response completed with error" err.code=unknown err.detail="filesystem: invalid path: /docker/registry/v2/repositories/smol/_manifests/tags/ssha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf/current/link" err.message="unknown error" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=1d4228f0-aa55-46db-b61e-c24099023b93 http.request.method=GET http.request.remoteaddr="172.17.0.1:32796" http.request.uri="/v2/smol/manifests/ssha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf" http.request.useragent="curl/7.64.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.4869ms http.response.status=500 http.response.written=244 vars.name=smol vars.reference="ssha256:9cb6208bf9dbdd2c776c64e8622f2dfed9f3426348d26dc6f86bb56b1261dcdf" 
172.17.0.1 - - [11/Mar/2020:10:30:15 +0000] "GET /v2/smol/manifests/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d HTTP/1.1" 500 80 "" "curl/7.64.1"
time="2020-03-11T10:30:15.3034493Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid character '\x1f' looking for beginning of value" err.message="unknown error" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=35686848-c8e2-4788-8e8f-bc17f80b1fe7 http.request.method=GET http.request.remoteaddr="172.17.0.1:32798" http.request.uri="/v2/smol/manifests/sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d" http.request.useragent="curl/7.64.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.2668ms http.response.status=500 http.response.written=80 vars.name=smol vars.reference="sha256:3ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771d" 
172.17.0.1 - - [11/Mar/2020:10:30:30 +0000] "GET /v2/smol/manifests/sha256:4ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771e HTTP/1.1" 404 180 "" "curl/7.64.1"
time="2020-03-11T10:30:30.9356156Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=smol revision=sha256:4ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771e" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=97dff837-485e-49fe-a65a-8923cf5266ab http.request.method=GET http.request.remoteaddr="172.17.0.1:32800" http.request.uri="/v2/smol/manifests/sha256:4ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771e" http.request.useragent="curl/7.64.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.1776ms http.response.status=404 http.response.written=180 vars.name=smol vars.reference="sha256:4ecd1c3829ce622c9e59853f48ec741441cd5c35ece50996c78eace0513e771e"

What I expected

Requesting the image config digest or a layer digest as a manifest should 404 because there is no manifest of that digest.

[deleteriousEffect]

I think this is caused by a fix for a bug introduced in 2.1.0 https://github.com/distribution/distribution/blob/main/registry/storage/registry.go#L222

I've looked for an issue in the past, but I couldn't find anything. From reading that comment, it seems like that 2.1.0 was linking manifests into _layers instead of _manifests, so we also have to look there. If we find something in the _layers directory, we try to parse it as json when getting as a manifest. Not sure how we could know for sure if something is a blob or a malformed/corrupted manifest, however.

[brackendawson]

Thanks for finding the root cause!

In my PR I check that what was retrieved is a manifest, but this would suppress a "correct" 500 error if a manifest was corrupted in storage and return a 404 instead, which is undesirable.

I wonder if the same check could be applied, but only to objects retrieved by the workaround (https://github.com/distribution/distribution/blob/main/registry/storage/registry.go#L222). Meaning we could continue to 200/404/500 as intended for manifests retrieved properly, but we would not be able to 500 a corrupt manifest retrieved using this workaround.

[deleteriousEffect]

I'm curious if we could remove the workaround. It's been six years since 2.1.0 was released and 2.1.1 was released the next day, resolving the issue with this PR: 44c7fb9

I can't imagine there are many, if any, manifests that are still in use that were affected by this issue.

[brackendawson]

I'd support that. It would definitely need a release note.

I'll also do some work to check the https://icr.io storage and see if we have any.

[brackendawson]

I checked and we never ran 2.1.0. I now also have strong doubts that any existing registries ever ran that version.

[deleteriousEffect]

Looks like we started with v2.3.1, so we shouldn't have been affected either.

[aaronlehmann]

I ran into this too. It really seems like incorrect behavior - the registry should return 404 in these cases. It triggered our alerting because 500 errors are usually a sign that something's wrong.

[milosgajdos]

Should we close this @brackendawson now that #3365 has been merged?

[brackendawson]

I think so, yes. Fixed in 3.0