.github: Pin docker buildx version to v0.9.1 #23206

joestringer · 2023-01-20T00:18:07Z

Somewhere between the combination of GitHub action
docker/build-push-action v3.3.0 and Docker buildx version v0.10.0,
provenance attestation was transparently added into the build process
for new images. Unfortunately, since we already have SBOM generation
steps in our workflows, this would break the workflows. The existing
workflows would attempt to pull the images with provenance and then
generate an SBOM from that existing attestation. This would lead to a
message like the following in CI image builds:

level=fatal msg="generating doc: creating SPDX document: generating                                                                                                                                                                                                                    
SPDX package from image ref quay.io/cilium/docker-plugin-ci:XXX:                                                                                                                                                                                                                       
generating image package"

I tried disabling provenance in the docker/build-push-action, but
apparently it just ignored such requests and pushed the attestation into
the image anyway. So, this commit attempts to revert buildx back to
v0.9.1 to prevent it from generating those artifacts.

This is a quick-and-dirty hack to stabilize CI for the short term, then
we can figure out over time how to properly resolve the conflict between
these systems.

joestringer · 2023-01-20T00:23:25Z

From @aanm:

good build run -> https://github.com/cilium/cilium/actions/runs/3962486446/jobs/6790837010#step:21:121

/usr/bin/docker buildx version
  github.com/docker/buildx 0.9.1+azure-2 ed00243a0ce2a0aee75311b06e32d33b44729689
/usr/bin/docker buildx build --build-arg OPERATOR_VARIANT=docker-plugin --file ./images/cilium-docker-plugin/Dockerfile --iidfile /tmp/docker-build-push-dpxY2M/iidfile --platform linux/amd64,linux/arm64 --tag quay.io/cilium/docker-plugin-ci:b6deb5d3f0a134221ab9ac5529bf564884e9cf83 --target release --metadata-file /tmp/docker-build-push-dpxY2M/metadata-file --push .

bad build run -> https://github.com/cilium/cilium/actions/runs/3963317099/jobs/6791009401#step:21:120

  /usr/bin/docker buildx version
  github.com/docker/buildx 0.10.0+azure-1 876462897612d36679153c3414f7689626251501
/usr/bin/docker buildx build --build-arg OPERATOR_VARIANT=docker-plugin --file ./images/cilium-docker-plugin/Dockerfile --iidfile /tmp/docker-build-push-WnGG1h/iidfile --platform linux/amd64,linux/arm64 --provenance mode=max,builder-id=https://github.com/cilium/cilium/actions/runs/3963317099 --tag quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728 --target release --metadata-file /tmp/docker-build-push-WnGG1h/metadata-file --push .

good image:

$ docker buildx imagetools inspect quay.io/cilium/docker-plugin-ci:9a58f44c43110e12f2300effba53fbdef68356d2
Name:      quay.io/cilium/docker-plugin-ci:9a58f44c43110e12f2300effba53fbdef68356d2
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:737e990cae2e4ccfea992de2ab2ce54c052c6af29a49f18689a1860082a8caed

Manifests:
  Name:      quay.io/cilium/docker-plugin-ci:9a58f44c43110e12f2300effba53fbdef68356d2@sha256:9a26c82f3baf480fab603254b6b642be7b4bfb71aa317d6a3af21b6b68e8daa5
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      quay.io/cilium/docker-plugin-ci:9a58f44c43110e12f2300effba53fbdef68356d2@sha256:df624ccbd55dd22cdbc6ed2cc2d8ed37904db6e86a0ae201e3ad66f827c1670c
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

bad image:

$ docker buildx imagetools inspect quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728
Name:      quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:0c32f1fb018d1c2a3f93cbf1c90ed291d790cfe1537a1cda8762ead72ec158b6

Manifests:
  Name:      quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728@sha256:c36a9e6d787d6a777706b654012cb7b45071b1ad45c94bdf759cba0d47fdbdd9
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/amd64

  Name:      quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728@sha256:7eb9061c13fa872687815b4c9f02c29de970a3ba5749910dd5775914777d6ce0
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm64

  Name:      quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728@sha256:9196f6add5c1d74918dddc4119143859996ec402d2744edfb8dd0b5f5fbfbd20
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  unknown/unknown
    vnd.docker.reference.digest: sha256:c36a9e6d787d6a777706b654012cb7b45071b1ad45c94bdf759cba0d47fdbdd9
    vnd.docker.reference.type:   attestation-manifest

  Name:      quay.io/cilium/docker-plugin-ci:0bc41809ca4a1b4b633bb111b99b492e33bc6728@sha256:57513aba6990656f8f328242afa2966eafae2cdd75a5d170ce200f218e70cc37
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  unknown/unknown
    vnd.docker.reference.digest: sha256:7eb9061c13fa872687815b4c9f02c29de970a3ba5749910dd5775914777d6ce0
    vnd.docker.reference.type:   attestation-manifest

joestringer · 2023-01-20T00:24:51Z

Failed with buildx v0.10.0: https://github.com/cilium/cilium/actions/runs/3963537803/jobs/6791477015

Looks like attestation landed in the image despite turning it off:

$ docker buildx imagetools inspect quay.io/cilium/docker-plugin-ci:12251ba7237325d24647805616b3bfba9ee3e635
Name:      quay.io/cilium/docker-plugin-ci:12251ba7237325d24647805616b3bfba9ee3e635
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:bc613250749ca24d9e88376de320687cc0fd5e7cd54a62515896be11611be231
           
Manifests: 
  Name:      quay.io/cilium/docker-plugin-ci:12251ba7237325d24647805616b3bfba9ee3e635@sha256:48ea8884489b8766895482cfc94150d09812e9ddef42d71a1e4e4f4b76da0de9
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/amd64
             
  Name:      quay.io/cilium/docker-plugin-ci:12251ba7237325d24647805616b3bfba9ee3e635@sha256:efba4deab9ff9507197a76a340dad62cf86db1a1db53113b71ee1f07691ea2aa
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm64
             
  Name:      quay.io/cilium/docker-plugin-ci:12251ba7237325d24647805616b3bfba9ee3e635@sha256:e351296d4cd684e39b4e4ef1ba7d260fa8573a892300c4aeb9e5c86a5c4b70f2
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  unknown/unknown
    vnd.docker.reference.digest: sha256:48ea8884489b8766895482cfc94150d09812e9ddef42d71a1e4e4f4b76da0de9
    vnd.docker.reference.type:   attestation-manifest
             
  Name:      quay.io/cilium/docker-plugin-ci:12251ba7237325d24647805616b3bfba9ee3e635@sha256:cf00f08408756916a1cf30071356a5ac83faf400f009c506b17408d410197a47
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  unknown/unknown
    vnd.docker.reference.digest: sha256:efba4deab9ff9507197a76a340dad62cf86db1a1db53113b71ee1f07691ea2aa
    vnd.docker.reference.type:   attestation-manifest

EDIT: We can see in the workflow file from the same run that provenance should be disabled:

https://github.com/cilium/cilium/actions/runs/3963537803/workflow

joestringer · 2023-01-20T00:33:42Z

build-images-ci is using pull_request_target, pull request target is documented as:

This event runs in the context of the base of the pull request, rather than in the context of the merge commit

So I guess the changes in this PR are ignored by CI.

joestringer · 2023-01-20T00:42:35Z

Latest failure: https://github.com/cilium/cilium/actions/runs/3963649065/jobs/6791721533

Again, the workflow file shows that provenance should be disabled, but it's not:
https://github.com/cilium/cilium/actions/runs/3963649065/workflow

🤷 either I'm messing up the configuration, or the provenance feature cannot be disabled by the upstream plugin. I'm going to try switching to an approach where we just pin the docker buildx version to an older version.

Somewhere between the combination of GitHub action docker/build-push-action v3.3.0 and Docker buildx version v0.10.0, provenance attestation was transparently added into the build process for new images. Unfortunately, since we already have SBOM generation steps in our workflows, this would break the workflows. The existing workflows would attempt to pull the images with provenance and then generate an SBOM from that existing attestation. This would lead to a message like the following in CI image builds: level=fatal msg="generating doc: creating SPDX document: generating SPDX package from image ref quay.io/cilium/docker-plugin-ci:XXX: generating image package" I tried disabling provenance in the docker/build-push-action, but apparently it just ignored such requests and pushed the attestation into the image anyway. So, this commit attempts to revert buildx back to v0.9.1 to prevent it from generating those artifacts. This is a quick-and-dirty hack to stabilize CI for the short term, then we can figure out over time how to properly resolve the conflict between these systems. Signed-off-by: Joe Stringer <joe@cilium.io>

joestringer · 2023-01-20T00:57:10Z

Looks like that did the trick, the CI action pulled docker buildx v0.9.1: https://github.com/cilium/cilium/actions/runs/3963731211/jobs/6791899664

michi-covalent

pin to win

sayboras

Docker buildx is just bumped to v0.10.0 today actions/runner-images#6941, so that probably explains mixed versions as part of transition.

nbusseneau · 2023-01-20T16:26:53Z

Pinning did not seem to fix the issue. We are removing SBOM in #23204 until we find a better fix. Should we revert this?

chancez · 2023-01-20T18:37:49Z

This isn't working because we set version on the wrong field. It doesn't go onto the build-push action, but the setup-buildx action.

joestringer · 2023-01-20T20:29:50Z

Dropping backport labels in favour of #23220 .

joestringer requested review from a team as code owners January 20, 2023 00:18

joestringer added the release-note/misc This PR makes changes that have no direct user impact. label Jan 20, 2023

joestringer requested review from aanm and tklauser January 20, 2023 00:18

maintainer-s-little-helper bot added dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. and removed dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels Jan 20, 2023

joestringer force-pushed the pr/joe/buildx-provenance branch from 12251ba to 0d6ebe1 Compare January 20, 2023 00:38

joestringer added the dont-merge/preview-only Only for preview or testing, don't merge it. label Jan 20, 2023

joestringer force-pushed the pr/joe/buildx-provenance branch from 0d6ebe1 to 2076586 Compare January 20, 2023 00:54

joestringer changed the title ~~.github: Disable provenance in docker action~~ .github: Pin docker buildx version to v0.9.1 Jan 20, 2023

joestringer removed the dont-merge/preview-only Only for preview or testing, don't merge it. label Jan 20, 2023

joestringer mentioned this pull request Jan 20, 2023

Action started to push manifest indexes instead of images for a single platform docker/build-push-action#755

Closed

michi-covalent approved these changes Jan 20, 2023

View reviewed changes

christarazi approved these changes Jan 20, 2023

View reviewed changes

joestringer added needs-backport/1.11 needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch labels Jan 20, 2023

maintainer-s-little-helper bot added this to Needs backport from master in 1.11.13 Jan 20, 2023

maintainer-s-little-helper bot added this to Needs backport from master in 1.12.6 Jan 20, 2023

sayboras approved these changes Jan 20, 2023

View reviewed changes

aanm approved these changes Jan 20, 2023

View reviewed changes

aanm merged commit 30b036d into master Jan 20, 2023

aanm deleted the pr/joe/buildx-provenance branch January 20, 2023 09:10

ldelossa mentioned this pull request Jan 20, 2023

v1.12 backports 2023-01-13 (replacement) #23190

Closed

joestringer mentioned this pull request Jan 20, 2023

.github: Pin docker buildx version to v0.9.1 (v2) #23220

Merged

joestringer removed needs-backport/1.11 needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch labels Jan 20, 2023

maintainer-s-little-helper bot removed this from Needs backport from master in 1.12.6 Jan 20, 2023

maintainer-s-little-helper bot removed this from Needs backport from master in 1.11.13 Jan 20, 2023

joestringer added release-note/ci This PR makes changes to the CI. and removed release-note/misc This PR makes changes that have no direct user impact. labels Jan 20, 2023

darthShadow mentioned this pull request Jan 20, 2023

Error Could not do a head request for containrrr/watchtower#1528

Closed

This was referenced Jan 20, 2023

Unable to update docker (with watchtower) pi-hole/docker-pi-hole#1289

Closed

Unable to update with watchtower PrivateBin/docker-nginx-fpm-alpine#137

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.github: Pin docker buildx version to v0.9.1 #23206

.github: Pin docker buildx version to v0.9.1 #23206

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023

michi-covalent left a comment

sayboras left a comment

nbusseneau commented Jan 20, 2023

chancez commented Jan 20, 2023

joestringer commented Jan 20, 2023

.github: Pin docker buildx version to v0.9.1 #23206

.github: Pin docker buildx version to v0.9.1 #23206

Conversation

joestringer commented Jan 20, 2023 • edited

joestringer commented Jan 20, 2023

joestringer commented Jan 20, 2023 • edited

joestringer commented Jan 20, 2023 • edited

joestringer commented Jan 20, 2023 • edited

joestringer commented Jan 20, 2023

michi-covalent left a comment

Choose a reason for hiding this comment

sayboras left a comment

Choose a reason for hiding this comment

nbusseneau commented Jan 20, 2023

chancez commented Jan 20, 2023

joestringer commented Jan 20, 2023

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023 •

edited

joestringer commented Jan 20, 2023 •

edited