`oci-artifact=true` should also affect the attestations at the image index

[polarathene]

Description

Original reference: moby/buildkit#5573 (comment)

When I opt-in to oci-artifact=true, the attestations are published as OCI 1.1 compliant, but the image index still has the Docker specific workaround instead of also being OCI 1.1 compliant? Should it not also have been adapted to OCI 1.1?

# NOTE: `--attest type=provenance,mode=min` is implicit by default
docker buildx build \
  --output type=image,push=true,oci-mediatypes=true,oci-artifact=true \
  --platform linux/amd64,linux/arm64 \
  --tag ghcr.io/polarathene/example:test \
  .

Actual Behaviour

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6f81b18808466808136cd43e68a156f7a58937bd4e50edacce158ac5300cbce5",
      "size": 668,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:c921e8c46326db1dbd537eaf8d9566408497e105d20a5f05baeaab09afff54b4",
      "size": 668,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6f1a7b24844a8ff6314eaa7fe99432de112731ef4e3a3811f58f07e062beea2d",
      "size": 914,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:6f81b18808466808136cd43e68a156f7a58937bd4e50edacce158ac5300cbce5",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:47fe6f33b043ac8b3999f3d76b4d5cc46a8c4dcff4a4eef5f1a9e826ef96c988",
      "size": 914,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:c921e8c46326db1dbd537eaf8d9566408497e105d20a5f05baeaab09afff54b4",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

Expected Behaviour

Expected image index should no longer reference manifests for attestation artifacts in the manifests list:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6f81b18808466808136cd43e68a156f7a58937bd4e50edacce158ac5300cbce5",
      "size": 668,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:c921e8c46326db1dbd537eaf8d9566408497e105d20a5f05baeaab09afff54b4",
      "size": 668,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

When publishing to a registry that is OCI v1.1 compliant with support for the Referrer API (or tag schema fallback), using oci-artifact=true publishes compatible OCI artifacts which can then be discovered.

This makes the non-compliant workaround specific to Docker redundant? The linked docs provide no context on compatible tooling that is reliant upon the data for artifact traversal/discovery (but lacks OCI v1.1 compatibility).

[tonistiigi]

No, that was not intended in moby/buildkit#5573 .

[polarathene]

@tonistiigi I have now looked into this rather extensively and would like to know why oci-artifact=true should not omit the attestations from the image index?

I've updated my original issue description expectation.

• Image index is documented to include the attestations to support unspecified tooling when using Docker media types instead of OCI + OCI artifacts.
• This docker specific workaround is not compatible with various registries that detect it as an unknown/unknown platform to pull.

If it serves no purpose with oci-mediatypes=true,oci-artifact=true, it should also be excluded (or an additional opt-out to allow this).

[tonistiigi]

The attestations are in index because they are part of the build result and so that tooling can find the attestation for matching image manifest and also for result equality check.

#5573 changed the attestation manifest to use the artifact manifest definition from OCI image-spec instead of using image manifest with fake config. It doesn't change anything about how attestations are linked to the image or their distribution model when pushing. Referrers API is part of distribution spec and there is no intention from BuildKit to start using it atm.

This docker specific workaround is not compatible with various registries that detect it as an unknown/unknown platform to pull.

The index produced by buildkit is a valid image index.

[polarathene]

so that tooling can find the attestation for matching image manifest and also for result equality check.

What tooling? The ambiguity does not help.

It doesn't change anything about how attestations are linked to the image or their distribution model when pushing.
Referrers API is part of distribution spec and there is no intention from BuildKit to start using it atm.

Is BuildKit the only tooling relying on this information? I assume maybe some Docker CLI commands or Docker Desktop features might also?

When images are built and published via other tooling that doesn't use BuildKit, registries will not have this additional metadata to discover the attestation artifacts associated to an image. However they would via the Referrers API when both the client and registry are compatible.

If all known tooling dependent upon the Docker vendored annotations added at the image index is Go-based, couldn't these tools leverage the Referrer API support from an existing Go project/library? ORAS has this functionality via oras discover for example.

I'm just trying to understand what existing tooling will become incompatible if I exclude this information from the image index. I'd rather not have the misleading unknown/unknown platform at a registry like GHCR, but I would like the attestations... yet the decision to deny opt-out complicates that.

---

This docker specific workaround is not compatible with various registries that detect it as an unknown/unknown platform to pull.

The index produced by buildkit is a valid image index.

Misunderstanding? The image index is "valid"... but registries don't treat it in a way that is considered acceptable (I'm sure you're aware of all the issues related to this?)

I'm referring to this:

The registries should not be displaying these entries as unknown/unknown is an invalid platform. However the image index is bundling these references to artifact manifests with references to image manifests, the registries don't know that difference when displaying such. I assume this occurs on other registries as well that aren't DockerHub.

Gitea also had this with their registry, only recently adding logic to hide the invalid platform. It's been an issue for years since provenance was introduced as a default part of builds, registries like GHCR are not accommodating like Gitea (which only did so in Dec 2024). It is misleading information to users, so some projects just avoid attaching attestations to prevent it.

---

What are the steps to have attestations without the above issue? Without an opt-out, I'd need to do something like export with type=oci,dest=./oci-image-layout, then parse that content to create a new image index without the attestation references?

I've not looked into other changes that might require from that modification since the digest would change for the image index JSON I'd need to accommodate for that, and then I can push it to the registry, or I try some similar dance to modify the image index published to the registry instead of the OCI image export workaround.

[tonistiigi]

What tooling? The ambiguity does not help.
I'm just trying to understand what existing tooling will become incompatible if I exclude this information from the image index.

Docker buildx imagetools, Scout, Desktop Build UI, engine, all provide ways for working with attestation data in images atm. Any tool for moving images in registry can also move these attestations atm.

When images are built and published via other tooling that doesn't use BuildKit,

This is in no way limited to Docker, anyone can integrate with our images and implement attestation lookup via annotations.

If all known tooling dependent upon the Docker vendored annotations added at the image index is Go-based, couldn't these tools leverage the Referrer API support from an existing Go project/library?

We don't just use the attestation descriptors for backwards compatibility with existing tooling but because we find it superior for our use case. I don't want to stir up any old drama, but we participated in what eventually became Referrers API and made it clear that it didn't work for us but were unfortunately ignored in the end.

I'm referring to this:

This is a question to the implementors of that UI, if the attestation manifests should be visible there (because they do take up space, and someone may want to access them directly), or hidden by default, or if the UI should show some extra "attestation" indicator etc.. The attestation manifest descriptors have unique annotation so it is easy to skip them in UI if that is desired. Web-UI is different from the registry. Eg. if you look at the same view in Docker Hub then they are hidden by default.

What are the steps to have attestations without the above issue?
then parse that content to create a new image index without the attestation references?

You can disable attestations for your build, but BuildKit does not support any other distribution method for attestations that is outside of the build result.

[polarathene]

When images are built and published via other tooling that doesn't use BuildKit,

This is in no way limited to Docker, anyone can integrate with our images and implement attestation lookup via annotations.

I meant if I used something else to build and publish an image that doesn't add these vendored annotations to the image index (desired to avoid the registry UX issue), all the tooling you cited that relies on this method to discover attestations would not be compatible correct?

So it works both ways? In that all that tooling you've cited could instead implement OCI artifact support as per the v1.1 OCI spec with the Referrer API?

---

but because we find it superior for our use case.

we participated in what eventually became Referrers API and made it clear that it didn't work for us but were unfortunately ignored in the end.

No worries, I can respect that 👍

It would be great if that could be clarified on the related docs, along with known tooling, since I wouldn't be surprised if images built and published via other tooling lacks this information and users relying on the tools you've cited being dependent upon that would not be a good UX? (unless these tools will offer a fallback to use Referrer API, doesn't sound like it though)

My understanding with oci-mediatypes=true,oci-artifact=true is that both are intended to become default in future, and the image index approach being used instead of Referrer API seems a tad hacky (I'd have thought OCI annotations on the respective images would have made more sense, but I'm sure there's a good reason that was avoided).

---

I'm referring to this:

This is a question to the implementors of that UI, if the attestation manifests should be visible there (because they do take up space, and someone may want to access them directly), or hidden by default, or if the UI should show some extra "attestation" indicator etc.. The attestation manifest descriptors have unique annotation so it is easy to skip them in UI if that is desired. Web-UI is different from the registry. Eg. if you look at the same view in Docker Hub then they are hidden by default.

I understand that. As a user of both buildx and ghcr.io, I can only reach out to both ends about the UX issue, but it seems ghcr (and other registries) are reluctant towards implementing conditional logic as a workaround on their end, while buildx is reluctant to allow opt-out on your end of it 🤷‍♂

Docker Hub hides the invalid platform usage because that's the registry associated to the bulk of tooling that prefers to keep this approach for referencing artifacts instead of Referrer API? It's not a surprise that it tailors for that, convincing other registries to do the same is not as easy 😓

---

What are the steps to have attestations without the above issue?
then parse that content to create a new image index without the attestation references?

You can disable attestations for your build, but BuildKit does not support any other distribution method for attestations that is outside of the build result.

I have built with attestations and pushed to zot, oras discover has no issue using Referrer API to discover the attestations as OCI artifacts. So I'm not sure what you mean by BuildKit not supporting other distribution methods for attestations, other than the intentional lack of opt-out for the enforced method.

This won't work however with GHCR which does not support the Referrer API presently, and buildx / BuildKit intentionally doesn't support the fallback with Referrer Tag schema. I'd have to manually do so separately via oras attach if I wanted that, or wait for GHCR to support the Referrer API.

---

Overall this is a minor UX issue on the surface, no notable problems beyond that.

Thanks for providing the extra context to clarify the compatibility situation better and what to expect going forward 🙏