BUILDX_NO_DEFAULT_ATTESTATIONS not respected after v5.3.0

[MarkusSintonen]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

We have been successfully building multi arch images with native GHA runners. But when updating build-push-action we noticed that BUILDX_NO_DEFAULT_ATTESTATIONS=1 no longer works in latest version of the action (v6.x). I was able to find that this got broken already in v5.3.0. We were previously using v5.0.0 so the jump to the latest version was quite large version bump.

(We on purpose do not use any QEMU based approaches for building the images. These are also private images we are building.)

Expected behaviour

docker manifest create should work in the example YAML workflow.

Actual behaviour

docker manifest create breaks with the cryptic error myregistry/myrepo:mytest-amd64 is a manifest list. This means that the arch specific images got some attestation data in them although BUILDX_NO_DEFAULT_ATTESTATIONS=1 was used as documented.

Repository URL

No response

Workflow run URL

No response

YAML workflow

name: Build multi arch image

on: workflow_call

jobs:
  test-multi-arch-build:
    strategy:
      matrix:
        include:
          - runner: ubuntu-latest
            suffix: amd64
          - runner: linux_ubuntu_24_04_arm64_2_cores
            suffix: arm64
    runs-on: ${{ matrix.runner }}
    env:
      BUILDX_NO_DEFAULT_ATTESTATIONS: 1  # Buildx added attestations are not compatible with "docker manifest" cmds
    steps:
      - uses: actions/checkout@v4.2.2

      - uses: docker/setup-buildx-action@v3.10.0

      - name: Build push image
        uses: docker/build-push-action@v5.3.0  # 5.3.0 is broken but 5.2.0 works. Also 6.x are broken.
        with:
          context: .
          tags: "myregistry/myrepo:mytest-${{ matrix.suffix }}"
          push: true

  test-multi-arch-manifest:
    needs: [ test-multi-arch-build ]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

      - name: Create Docker multi arch manifest
        env:
          IMAGE: "myregistry/myrepo:mytest"
        run: docker manifest create $IMAGE --amend $IMAGE-amd64 --amend $IMAGE-arm64

Workflow logs

Error: myregistry/myrepo:mytest-amd64 is a manifest list from docker manifest create

[crazy-max]

# 5.3.0 is broken but 5.2.0 works. Also 6.x are broken.

I looked at changes made between 5.2.0 and 5.3.0: v5.2.0...v5.3.0 and there is only a dependency change for the actions-toolkit: #1080 that doesn't have any attestation logic changes: docker/actions-toolkit@v0.18.0...v0.19.0 So I don't think that's true but if you can provide a repro with workflow logs running both versions we can take a look.

Workflow logs

We would need full logs of name: Build push image step to understand the underlying issue.

[MarkusSintonen]

So I don't think that's true but if you can provide a repro with workflow logs running both versions we can take a look.

When I just make single change from 5.2.0 to 5.3.0. It breaks with the mentioned error.

We would need full logs of name: Build push image step to understand the underlying issue.

Here with 5.3.0 (stripped some internal stuff from there):

logs.txt

[MarkusSintonen]

I can actually now see what is going wrong. In 5.2.0 its not adding the provenance args:

[command]/usr/bin/docker buildx build --iidfile /home/runner/work/_temp/docker-actions-toolkit-6ZT9EI/iidfile --tag myregistry/myrepo:mytest-arm64 --metadata-file /home/runner/work/_temp/docker-actions-toolkit-6ZT9EI/metadata-file --push myregistry/myrepo

But in 5.3.0 the --provenance appears eventhough env is set BUILDX_NO_DEFAULT_ATTESTATIONS=1:

[command]/usr/bin/docker buildx build --iidfile /home/runner/work/_temp/docker-actions-toolkit-yApl8C/iidfile --provenance mode=min,inline-only=true,builder-id=xxx --tag myregistry/myrepo:mytest-arm64 --metadata-file /home/runner/work/_temp/docker-actions-toolkit-yApl8C/metadata-file --push myregistry/myrepo

[crazy-max]

Here with 5.3.0 (stripped some internal stuff from there):

logs.txt

Thanks I'm taking a look

[crazy-max]

I was able to repro.

With 5.2.0 the manifest looks like this:

{
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "digest": "sha256:81653b327209a2e5cb568d1490011c51dc802d928ac72db5becb9992d4d78a4c",
  "size": 695
}

But with 5.3.0 we also have the attestation:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "digest": "sha256:e40c1709e89261b4423543fd15796027990017d2ea33dbabd793dbeb778eeee9",
  "size": 855,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:09704d78087e896529f3d0047cd336e4c7376c5eac523f31d624bb1468b74c86",
      "size": 667,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:a07c38d9363f8001e99d0fe2db880c89f6123aa8b1e0ff25c59a8d26810c13f6",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:09704d78087e896529f3d0047cd336e4c7376c5eac523f31d624bb1468b74c86",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

I'm investigating on this issue and keep you posted.

[crazy-max]

It looks related to this change in our actions-toolkit docker/actions-toolkit@v0.18.0...v0.19.0#diff-52c8c47c24f59e49e8d73143d55935e281ac179a8483b9fd8a53a479a524885fR135-R136 where buildkit version was not detected correctly and therefore in build-push-action:

build-push-action/src/context.ts

Line 267 in 84ad562

} else if (!hasAttestProvenance && (await toolkit.buildkit.versionSatisfies(inputs.builder, '>=0.11.0')) && !Build.hasDockerExporter(inputs.outputs, inputs.load)) {

This condition is satisfied because BuildKit >= 0.11.0 so this is working as expected but I think we should also take into account BUILDX_NO_DEFAULT_ATTESTATIONS. Working on it.