[Bug]: manifest download with digest reference fails with 500 'image meta not found'

[akashsinghal]

zot version

v2.0.2-rc.2

Describe the bug

I'm trying to download a manifest using a digested reference: localhost:5000/cosign@sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609 but it fails with a 500 error. Looking at the zot logs I can see

{"level":"error","error":"image meta not found","repository":"cosign","reference":"sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609","goroutine":195,"caller":"zotregistry.dev/zot/pkg/meta/hooks.go:126","time":"2024-03-06T19:45:32.029070934Z","message":"failed to update stats on download image"}
{"level":"info","module":"http","username":"test_user","component":"session","clientIP":"172.17.0.1:38856","method":"GET","path":"/v2/cosign/manifests/sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609","statusCode":500,"latency":"0s","bodySize":0,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["oras/1.1.0"]},"goroutine":195,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-03-06T19:45:32.029117534Z","message":"HTTP API"}

I've pushed an image localhost:5000/cosign:signed-key and used latest cosign to sign the image. I'm using oras manifest fetch to download the manifest of the cosign index. I can download the manifest if I reference it by tag but not by digest directly.

I've confirmed that the digest of the cosign index is correct:

> oras manifest fetch --descriptor localhost:5000/cosign:sha256-4790bc62b2852dd3a5f76e1b1866d3895d1125a487365c67e1ebee86150f6a67.sig
{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609","size":558}

> oras manifest fetch localhost:5000/cosign:sha256-4790bc62b2852dd3a5f76e1b1866d3895d1125a487365c67e1ebee86150f6a67.sig | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 233,
    "digest": "sha256:90f8b702351dacacb43a9a136f777ced4667d3218cf6a336a843c0d4d7bc00b7"
  },
  "layers": [
    {
      "mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
      "size": 237,
      "digest": "sha256:e04a3b6d1521632c6033b6ffeb867bc142f6f7227650feadf073bbb339267836",
      "annotations": {
        "dev.cosignproject.cosign/signature": "<SIG CONTENTS>"
      }
    }
  ]
}

> oras manifest fetch localhost:5000/cosign@sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609
Error: failed to fetch the content of "localhost:5000/cosign@sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609": GET "http://localhost:5000/v2/cosign/manifests/sha256:f092da75cf80b71e7244f64b76bcd8f3cfebda0951fd00379bea2c8289379609": response status code 500: Internal Server Error

To reproduce

Configuration

{
    "Storage": {
      "RootDirectory": "/var/lib/registry"
    },
    "HTTP": {
      "Address": "0.0.0.0",
      "Port": "5000",
      "Auth": {
        "HTPasswd": {
          "Path": "/etc/zot/htpasswd"
        }
      }
    }
}

Full Logs: https://gist.github.com/akashsinghal/f5f9705c925a3b375a66f181e13f2156

Expected behavior

No response

Screenshots

No response

Additional context

No response

[rchincha]

@akashsinghal thanks for filing the issue.

https://github.com/sigstore/cosign/releases/tag/v2.2.3
^ is this the cosign version used?

[akashsinghal]

@akashsinghal thanks for filing the issue.

https://github.com/sigstore/cosign/releases/tag/v2.2.3 ^ is this the cosign version used?

@rchincha Yes that's correct. Also I tried using zot 2.0.1 and same issue there. ORAS is 1.1.0

[rchincha]

@akashsinghal ok, able to repro this.

skopeo copy --format=oci docker://alpine:edge docker://localhost:5000
cosign generate-key-pair
cosign sign --key=cosign.key localhost:5000/alpine:edge
oras manifest fetch localhost:5000/alpine@<sha256-of-signature-itself>

[andaaron]

This bug doesn't reproduce if cosign uses an actual artifact type for the signature.

export COSIGN_EXPERIMENTAL=1
cosign sign --registry-referrers-mode=oci-1-1 [...]

Ideally this behavior should have been the default by now, since the 1.1 spec has been released. It is no longer an experiment.
Either this or continue pulling the signatures by the tag...

[akashsinghal]

Curious, what is the root cause of the issue here? Is cosign doing something unique or is it that any digest-based reference will fail for a manifest download? As a sanity check, I tried pushing another OCI image (no cosign in the picture here) by tag and then fetched by its digest reference and that worked fine.

[andaaron]

We have special handling for signatures, we save the data differently in the DB.

Given pull was made by digest (and we don't know it is supposed to be a signatured based on the reference) and the manifest created by cosign this way doesn't have an artifact type, and the config mediatype is the same as for container images, zot considered it a container image and tries looking for it in the DB as an image, resulting in an error.

If you pull it my tag zot considers it to be a signature based on the tag naming.
If you had signed using --registry-referrers-mode=oci-1-1 the manifest itself has a config with a mediatype specific to cosign, so we know it's a signature.

[akashsinghal]

We have special handling for signatures, we save the data differently in the DB.

Given pull was made by digest (and we don't know it is supposed to be a signatured based on the reference) and the manifest created by cosign this way doesn't have an artifact type, and the config mediatype is the same as for container images, zot considered it a container image and tries looking for it in the DB as an image, resulting in an error.

If you pull it my tag zot considers it to be a signature based on the tag naming. If you had signed using --registry-referrers-mode=oci-1-1 the manifest itself has a config with a mediatype specific to cosign, so we know it's a signature.

Thanks for the explanation. Makes sense now why it'll work for OCI 1.1 supported Cosign

[rchincha]

@akashsinghal we take OCI conformance very seriously.
That said, we should fix this bug.

[rchincha]

This bug doesn't reproduce if cosign uses an actual artifact type for the signature.

@akashsinghal just curious, for your use case, are you unable to move to cosign with OCI 1.1 referrers support?

[andaaron]

Fixed in #2301