`NotFound: content digest sha256:xxxx: not found` error when building cross platform images

[anze3db]

Description

On my M3 Mac I am unable to build build a docker image if I use --label and --platform linux/amd64 parameters when using the old build backend.

Reproduce

1. Create a Dockerfile:

FROM busybox

2. Build

DOCKER_BUILDKIT=0 docker build \
--label io.prefect.version=2.8.1 \
--platform linux/amd64 \
.

Expected behavior

The image should be successfully built, but instead I get the following error NotFound: content digest error:

❯ DOCKER_BUILDKIT=0 docker build \
--label io.prefect.version \
--platform linux/amd64 \
.
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  4.306MB
Step 1/2 : FROM busybox
 ---> 498a000f370d
Step 2/2 : LABEL io.prefect.version=2.8.1
NotFound: content digest sha256:bc17845ef50fdfe6643ea6535eb9df68623dc52c7a3bceb8401fe85d9dca2a0e: not found

docker version

Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:37:19 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.38.0 (181591)
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:25 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.5.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.7.3
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.1-desktop.2
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4-desktop.1
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.4
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.16.1
    Path:     /Users/anzepecar/.docker/cli-plugins/docker-scout

Server:
 Containers: 5
  Running: 1
  Paused: 0
  Stopped: 4
 Images: 21
 Server Version: 27.5.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.1.12-0-g51d5e946
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.12.5-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 14
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: ecb099ae-1ab5-49f7-bf44-74d26041a13f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/anzepecar/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

1868CD3D-DA87-42B8-8AC0-F0B20CFA1675/20250228191005

Additional Info

There are a few ways to mitigate this issue:

1. Use buildkit
2. Remove the --label parameter
3. Remove --platform parameter, build the image and then add the --platform parameter back. After this all future builds for this particular image will succeed.
4. Use a VPN to change your location. We noticed that this issue only shows up in certain regions (confirmed broken around Lisbon, confirmed working around San Francisco) and I was able to resolve the problem by switching to a different location with a VPN 🤷‍♂️
5. Switch from Docker Desktop to OrbStack (the issue doesn't reproduce in OrbStack).

We encountered this issue when using Prefect to build the image (original issue on Prefect). Prefect uses docker-py, which doesn't support buildkit.

I understand that this problem likely won't get fixed since it's only reproducible on the deprecated build backend, but we have spent a fair amount of time debugging this issue internally. I'm hoping this issue will save time to anyone else that stumbles into this problem!

[thaJeztah]

Thanks for reporting; my first inkling is that there's possibly some gaps in the legacy builder implementation when using the containerd image-store.

Switch from Docker Desktop to OrbStack (the issue doesn't reproduce in OrbStack).

When using OrbStack, is the docker engine running with the containerd image store enabled, or is it using a graph-driver (non multi-arch) image store?

With the containerd image store enabled, the docker info output would show something like;

 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

With graphdriver, it would look something like;

Storage Driver: overlay2
  ...

[anze3db]

@thaJeztah When running OrbStack docker info says:

 Storage Driver: overlay2

Let me know if I can share any additional info, thanks!

[thaJeztah]

Thanks! So, yes, that confirms my suspicion that there's an issue in the implementation of the legacy builder when using the multi-arch containerd image store.

That's something that should be looked into! I'll see i someone is able to reproduce the issue. I see you're currently on a version of Docker Desktop with Docker Engine v27.5.x - we did a v28.,0 release (included in latest desktop builds), so we also need to check if perhaps the problem has been resolved in that version.

In the meantime, and this is mostly a workaround, if this is impacting you, you can try if switching Docker Desktop back to use the graph drivers fixes the immediate issue. When doing so, I'd recommend deleting your images and containers before switching (the storage locations are separate, and switching between stores does not purge the data, which would continue to occupy space).

You can disable the containerd image store in the "General" settings in Docker Desktop (see https://docs.docker.com/desktop/features/containerd/)

I should add that disabling the containerd image store will limit some features, e.g. when building images with BuildKit, it won't be able to store SBOMS / Attestations, and it's not possible to store multi-platform images locally (or more factually; multiple platform variants for the same image).

cc @vvoland

[anze3db]

@thaJeztah I've updated to the latest Docker Desktop version (Docker Engine 28.0.1) and I can confirm that this has fixed the issue!

There is a warning during the build that wasn't there before (The requested image's platform does not match the detected host - I think this makes sense since, but odd that it's appearing during the LABEL step), but the build finishes successfully:

DOCKER_BUILDKIT=0 docker ruby \
--label io.prefect.version=2.8.1 \
--platform linux/amd64 \
.
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  4.307MB
Step 1/2 : FROM ruby
 ---> d780a9b44dc5
Step 2/2 : LABEL io.prefect.version=2.8.1
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 86178d931bfe
 ---> Removed intermediate container 86178d931bfe
 ---> 9775edf1c1d2
Successfully built 9775edf1c1d2

I'm going to double check with my coworkers to make sure they also can't reproduce the problem anymore, but I think we can close this issue as fixed. 🎉

[anze3db]

Nobody on my team can reproduce this on Docker Engine 28.0.1 so I'm closing the issue. Thanks!