'buildx imagetools create' broken since v0.31.1 when building multi-platform images on github across multiple runners

[yubiuser]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

We've been using a github workflow which basically follows the steps outlined in

https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

Our workflow file is here.

We noticed, the 'merge' step failed since 6th of February, which was the day docker/setup-buildx-action started to deploy buildx version v0.31.1.
The logs always error at the docker buildx imagetools create step with

#1 [internal] pushing ghcr.io/pi-hole/***
#1 24.69 pushing sha256:1ac73ae4246c36835675145362d146a4bd2af7bd518de3e2bda44cf62116b55d to ghcr.io/pi-hole/***:nightly
#1 DONE 25.1s
ERROR: ghcr.io/pi-hole/***@sha256:f4a9f4daa0088ae6b74c45149690f9c31420122ad9079c231df81b2e31629010: not found
Error: Process completed with exit code 1.

We've manually reverted to buildx v0.31.0 and it immediately started to work again.
Here is a github action run which failed
https://github.com/pi-hole/docker-pi-hole/actions/runs/22030313470/job/63654289108

and this one is right after the version was set to v0.31.0
https://github.com/pi-hole/docker-pi-hole/actions/runs/22039970323/job/63679202890

As v0.31.1 only contained a single PR, it's probably due to #3632

Expected behaviour

Following the official guide for creating multi-arch images across multiple runners should not fail at step docker buildx imagetools create

Actual behaviour

See above. Workflows fail at the docker buildx imagetools create step.

Buildx version

affected are all versions > 0.31.0

Docker info

Builders list

Not sure how to obtain at github action runners.

Configuration

This is the Dockerfile we build: https://github.com/pi-hole/docker-pi-hole/blob/master/src/Dockerfile
And this is the corresponding workflow: https://github.com/pi-hole/docker-pi-hole/blob/master/.github/workflows/build-and-publish.yml

Build logs

Additional info

No response

[thaJeztah]

cc @crazy-max @tonistiigi

[crazy-max]

Thanks for your report.

What changed in v0.31.1 is imagetools create copy behavior from #3623: it now copies only the selected descriptor(s), instead of relying on broader copy side effects. Your workflow runs imagetools create twice and reuses all metadata tags (docker hub + GHCR) for both runs; with this pattern, the second run can reference ghcr.io/...@sha256:... digests that are not present in GHCR yet, which matches the not found error you see.

So this is reproducible with that workflow shape after v0.31.1. The immediate workaround is to split tags per registry (Docker Hub tags for the Docker Hub create call, GHCR tags for the GHCR call), or otherwise ensure each create call uses source digests from the same registry it is targeting. Something similar to: https://github.com/crazy-max/docker-healthchecks/blob/eef77d758b6c8b06220adc3e4b4f11da1eca6a89/.github/workflows/build.yml#L188-L191

Also, if you want to simplify this flow, consider switching to our GitHub Builder. This keeps multi-platform image publishing to Docker Hub and GHCR while simplifying the repository-specific GitHub Actions logic. Also reduces CI workflow complexity and maintenance burden for multi-platform builds. Compared to our "Distribute build across multiple runners" docs, this removes the need for per-platform matrix orchestration, digest artifact handoff, and a separate manifest merge job.

If you're interested I pushed these changes on a branch: crazy-max/docker-pi-hole@7195273. There is just this build-arg PIHOLE_DOCKER_TAG=${{ steps.meta.outputs.version }} that would not work because metadata outputs are not exposed here but that probably something we could improve in our reusable workflows using handlebars templates.

[yubiuser]

Thanks for your comprehensive answer. You really went the extra mile!

I now see where and why we have the issue here. I can implement the workaround but your GitHub Builder looks way more interesting! It would free us from a huge maintenance burden, because handling and orchestrating those distributed builds was not fun. I'm looking forward for the implementation of

metadata outputs are not exposed

and then I'll switch.

[tonistiigi]

Some more summary for this issue in case anyone else hits something similar.

imagetools create has 3 possible ways it can create target image:

1. make a carbon-copy if single arg and no --platform
2. make a merged copy if multiple sources
3. make a filtered copy if --platform filter set.

The case that seems to be hit in here is merged copy where sources are image indexes. In this case the source image index is discarded and replaced with the merged index. Before v0.31.1 (possibly after --platform was added and before v0.31.1) we used to have a bug where we still accidentally copied the source index (and descriptors for unused platforms) as well, even though they remained unreferenced by the target image. This was fixed in v0.31.1 .

If there is a valid use case to access the source index but leave it untagged, we could consider an extra option that would allow push-by-digest when creating the target image.

@yubiuser If this makes sense then I think we can close this issue. If you think push-by-digest could be useful, you can open a new issue to track that.

[yubiuser]

I think we can close the issue. You have been more than helpful.
Once we switch to the GitHub Builder, we don't have a need for push-by-digest. This was only ever a mean to get things done, nothing we loved to deal with.

[crazy-max]

@yubiuser Opened docker/github-builder#142