[registry api incompatibility] wrong digests for some images on quay.io (old registry)

[Zebradil]

I bumped into the issue with quay.io/thanos/thanos:v0.8.1 (on the tags page, use filter by tag to find exactly that one).
kbld resolves wrong digest for that image. I tested another one from quay.io (prometheus/node-exporter) and its digest was resolved correctly.

Along with the tagged image I put another one with correct digest in the example below.

test.yaml:

---
kind: 'Object'
spec:
- name: thanos
  image: 'quay.io/thanos/thanos:v0.8.1'
- name: thanos-digest
  image: 'quay.io/thanos/thanos@sha256:e008f9f98a403d6e872baf4b97ca85e7be79d401a43c6f85cf5ad170f1c21646'

$ kbld -f test.yaml
resolve | final: quay.io/thanos/thanos@sha256:e008f9f98a403d6e872baf4b97ca85e7be79d401a43c6f85cf5ad170f1c21646 -> quay.io/thanos/thanos@sha256:e008f9f98a403d6e872baf4b97ca85e
7be79d401a43c6f85cf5ad170f1c21646
resolve | final: quay.io/thanos/thanos:v0.8.1 -> quay.io/thanos/thanos@sha256:d6bcedf93f1a2ef27f3a0c8dd8bfb6bd86e6ae89352fdbb79354fd59bce6fc1b
---
kind: Object
metadata:
  annotations:
    kbld.k14s.io/images: |
      - Metas:
        - Tag: v0.8.1
          Type: resolved
          URL: quay.io/thanos/thanos:v0.8.1
        URL: quay.io/thanos/thanos@sha256:d6bcedf93f1a2ef27f3a0c8dd8bfb6bd86e6ae89352fdbb79354fd59bce6fc1b
      - Metas: null
        URL: quay.io/thanos/thanos@sha256:e008f9f98a403d6e872baf4b97ca85e7be79d401a43c6f85cf5ad170f1c21646
spec:
- image: quay.io/thanos/thanos@sha256:d6bcedf93f1a2ef27f3a0c8dd8bfb6bd86e6ae89352fdbb79354fd59bce6fc1b
  name: thanos
- image: quay.io/thanos/thanos@sha256:e008f9f98a403d6e872baf4b97ca85e7be79d401a43c6f85cf5ad170f1c21646
  name: thanos-digest

Succeeded

After looking through the code I found out that kbld uses outdated dependency for digest resolution. The wrong digest is calculated by taking sha256 from image's manifest, whereas the correct digest is being sent in headers by registry.
It was fixed in google/go-containerregistry.

Just updating dependency is not enough — API is changed and with the new version kbld can't be built anymore. Unfortunately, I have a little understanding of this project and can't rapidly fix the API usage.

As a workaround I just use the correct digest instead of tag.

[cppforlife]

After looking through the code I found out that kbld uses outdated dependency for digest resolution. The wrong digest is calculated by taking sha256 from image's manifest, whereas the correct digest is being sent in headers by registry.
It was fixed in google/go-containerregistry.

Note the comment in the code in that change:

		// If we can parse the digest from the header, and it's a signed schema 1
		// manifest, let's use that for the digest to appease older registries.

quay.io has been "problematic" for various images. given that it seems that this is a quay.io implementation quirk and not part of the OCI registry spec, i am not too hot about changing kbld code that follows the spec. i hope it's not a huge problem for you.

[Zebradil]

Yes, I saw the comment.
It's not a problem for me, because of the workaround and we tend to store images in our own registries anyways, I used that image from quay.io only for testing purposes.

This issue might help other people to reduce time spent on investigation.

As a side note, I think it worth keeping dependencies somewhat up-to-date at least for security reasons.

Please, feel free to close the issue if you think so.

[cppforlife]

This issue might help other people to reduce time spent on investigation.

yeah, i was even considering at some point just blocking use of quay.io without explicit ImageOverrides. not sure if thats too much.

As a side note, I think it worth keeping dependencies somewhat up-to-date at least for security reasons.

i typically keep an eye on them and update if security related issues come up.

ill close this issue for now.