Default image output from buildx v0.10 cannot run on Google Cloud Run or AWS Lambda

[jedevc]

As of Buildx 0.10, images are built with provenance -- this behavior is opt-out, and can be disabled with --provenance=false if using the buildx build cli, or provenance: false if using docker/build-push-action.

Images built with these default attestations that use the attestation storage from buildkit cannot be run on Google Cloud Run or AWS Lambda. While both Cloud Run and Lambda support the OCI format, they do not support multi-platform images.

From Google Cloud Run docs (emphasis mine):

Cloud Run accepts container images in the Docker Image Manifest V2, Schema 1, Schema 2, and OCI image formats.

Manifest lists used for Multi-Architecture Images are not supported.

Note

Cloud Run has now added support for processing image indexes: #1533 (comment).

From Lambda docs (emphasis mine):

Lambda provides multi-architecture base images. However, the image you build for your function must target only one of the architectures. Lambda does not support functions that use multi-architecture container images.

Ideally, GCR and Lambda should support multi-platform images, and detect the current platform from the Docker manifest list / OCI index.

As a temporary workaround (in order of preference):

• Users should set --provenance=false on buildx build, or set provenance: false on docker/build-push-action
• Users can force an explicit buildx version to v0.9.1
• Users can force an explicit buildkit version to v0.10

[steren]

Cloud Run PM here, we received many reports about this. At the moment, Cloud Run doesn't support multi architecture images.

We were tracking this work item, but we do not have an delivery date to communicate yet.

[piotrekkr]

@steren If you are Cloud Run PM then I have a request that can help save few hours of debugging for everyone. Would be nice to return proper error message when we try to use multi-platform images with cloud run instead of returning

Image 'europe-docker.pkg.dev/xxx/xxxx' not found

It clearly exist in artifact registry since I choose it from select list when creating cloud run service.

Message like:

Image 'europe-docker.pkg.dev/xxx/xxxx'  is a multi-platform image and is not supported

Would probably reduce debug time to 15 min instead of 6h.

Thanks

[davideme]

@steren and buildx team, Cloud Run supports SLSA Build level 3 through provenance, see https://cloud.google.com/software-supply-chain-security/docs/sds/deploy-run-view-security-insights#build
Is there a way to support provenance in buildx without multi architecture images?
Or the way provenance is added by Google Cloud Build and buildx is not the same?

[billinghamj]

Cloud Run was supporting the old Docker "fat manifest" manifest lists application/vnd.docker.distribution.manifest.list.v2+json - we've been using them for over a year

It just doesn't support the new OCI format

[billinghamj]

I would also note that the Docker documentation seemingly isn't up to date on covering this change: https://docs.docker.com/registry/spec/manifest-v2-2/

[christallire]

I'm leaving this for anyone who google searching: This also broke image pull on EKS and ECR.

Failed to pull image "account-id.dkr.ecr.region.amazonaws.com/image:tag": rpc error: code = NotFound desc = failed to pull and unpack image "account-id.dkr.ecr.region.amazonaws.com/image:tag": no match for platform in manifest: not found

provenance: false works as an workaround.

We did some testing and found that we only had this issue on clusters running on version 1.21. All other versions (1.22, 1.23 and 1.24) all seemed to work fine for us.

I'm not sure, but I'm on v1.24.7-eks-fb459a0

[steren]

Thanks for your patience

Cloud Run now supports multi-architecture images as long as the manifest list includes amd64/linux.

We now expect buildx images built with default settings to deploy to Cloud Run, please let us know here if you still experience issues.

[hadim]

I tested building an image with provenance: true and using docker/build-push-action@v4. The built image is then pushed to GCR, but I still see the reported weird 3 images being shown on the GCR dashboard with outdated date and image size of 0 bytes.

Doing the same while reverting to provenance: false, works as it was before.

[steren]

Try pushing to Artifact Registry instead of Container Registry

[hadim]

Indeed it seems to work. Thanks.