Multi-platform docker run errors with "Cannot overwrite digest" when using image manifest #43188

marcosnils · 2022-01-25T20:26:17Z

Description

I've built a multi-platform image and I'm trying to use the image manifest digest to run the two platform versions locally and every time I'm getting an error on the second run.

Steps to reproduce the issue:

docker run --rm --platform linux/amd64 docker.io/marcosnils/hello:latest@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288
Unable to find image 'marcosnils/hello:latest@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288' locally
docker.io/marcosnils/hello@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288: Pulling from marcosnils/hello
59bf1c3509f3: Already exists
7b313383a6e2: Pull complete
6a1dfa013b7c: Pull complete
Digest: sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288
Status: Downloaded newer image for marcosnils/hello@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288
Hello world, GOOS=linux, GOARCH=amd64


docker run --rm --platform linux/arm64 docker.io/marcosnils/hello:latest@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288
Unable to find image 'marcosnils/hello:latest@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288' locally
docker.io/marcosnils/hello@sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288: Pulling from marcosnils/hello
9b3977197b4f: Already exists
7b313383a6e2: Pull complete
780387bce059: Pull complete
Digest: sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288
docker: Cannot overwrite digest sha256:0a671df5c5d380023e8c63653134695b0e427b2fce95cc2da6fa7187182f9288.
See 'docker run --help'.

Describe the results you received:

Getting Cannot overwrite digest $DIGEST after the second command

Describe the results you expected:

Container to run successfully for the specified platform

Additional information you deem important (e.g. issue happens only occasionally):

This issue doesn't happen if you use the image+tag combination and remove the image digest

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:42 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 9
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.13.0-27-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.06GiB
 Name: tp
 ID: TQIA:ROA4:HGGA:3AG4:42VW:5KZO:ZNCV:DBBV:2DLK:HZFK:ITMA:RYWG
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: marcosnils
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Linux tp 5.13.0-27-generic #29~20.04.1-Ubuntu SMP Fri Jan 14 00:32:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

thaJeztah · 2022-02-03T19:00:20Z

Thanks for opening! (Sorry for not replying sooner, I'll discuss this issue with some maintainers)

The Error itself comes from

moby/reference/store.go

Lines 157 to 160 in 7b9275c

    
           // force only works for tags 
        
           if digested, isDigest := ref.(reference.Canonical); isDigest { 
        
           	return errors.WithStack(conflictingTagError("Cannot overwrite digest " + digested.Digest().String())) 
        
           }

And (as described) this error only happens when pulling by digest; my initial thinking was that it could be related to manifest caching (so skipping some steps if it already has the manifest cached?)

honnix · 2022-04-13T10:11:12Z

Hi, we are hitting this issue as well. Is there any update? Thanks.

tommyknows · 2022-07-22T10:54:22Z

I have been inspecting something like this, and it seems to come down to how multi-platform images are handled. I'm still trying to wrap my head around this, but will try to summarise what I have found out so far.

When you pull an image, it has an associated digest:

$ docker pull ubuntu:22.10
22.10: Pulling from library/ubuntu
46465228a6c0: Pull complete
Digest: sha256:d2d9a7a7b18ecb6a33befb2ca9d386462721b6523f165985719756778046b254
Status: Downloaded newer image for ubuntu:22.10
docker.io/library/ubuntu:22.10

Now you can actually inspect the image manifest. I've trimmed some parts away to make it easier to follow.

docker manifest inspect ubuntu@sha256:d2d9a7a7b18ecb6a33befb2ca9d386462721b6523f165985719756778046b254
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 529,
         "digest": "sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 529,
         "digest": "sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      }
   ]
}

As far as I understand, the digest sha256:d2d9a7a7b18ecb6a33befb2ca9d386462721b6523f165985719756778046b254 matches this multi-platform image.
It's also possible to fetch these images now with their platform-specific digests:

$ docker pull ubuntu@sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4
docker.io/library/ubuntu@sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4: Pulling from library/ubuntu
415d72858c74: Pull complete
Digest: sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4
Status: Downloaded newer image for ubuntu@sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4
docker.io/library/ubuntu@sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4

Note that I'm on an ARM chip and in the above command I pulled the AMD64 image.

I can also pull the ARM image, and the missing "Pull complete" message indicates that it did not need to download anything:

$ docker pull ubuntu@sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b
docker.io/library/ubuntu@sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b: Pulling from library/ubuntu
Digest: sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b
Status: Downloaded newer image for ubuntu@sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b
docker.io/library/ubuntu@sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b

This is because it already downloaded that with the initial ubuntu:22.10 tag because it matches my architecture.

So we have three digests in here:

The "root" digest, sha256:d2d9a7a7b18ecb6a33befb2ca9d386462721b6523f165985719756778046b254, which is platform-independent
The AMD64 image digest, sha256:4a260e959a8e5186f1915e46bfb7663165ebfe5a7dc3365c76f9e3fb8d3019e4
The ARMv8 image digest sha256:a9bb8163533a936c7f4463319a299ff15e35a24db77ba6838d56c14f03b9863b

The issue you're hitting is that you're specifying the root digest, instead of the platform-specific digest. Because that root-digest already exists, but maps to an image for a different architecture, you get the error-message Cannot overwrite digest.

To work around this, you can use the platform-specific digests instead.

mcascone · 2022-12-14T16:36:34Z

I just ran into this starting kind on my M1 Mac. I had previously set DOCKER_DEFAULT_PLATFORM=linux/amd64 as a workaround to an issue where docker images built on M1 (and presumably M2) will not work on non-M1 unix/linux architecture.

> kind create cluster --name kindcluster
Creating cluster "kindcluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: command "docker run --name kindcluster-control-plane --hostname kindcluster-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=kindcluster --net kind --restart=on-failure:1 --init=false --publish=127.0.0.1:50181:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.25.3@sha256:f52781bc0d7a19fb6c405c2af83abfeb311f130707a0e219175677e366cc45d1" failed with error: exit status 125
Command Output: Unable to find image 'kindest/node:v1.25.3@sha256:f52781bc0d7a19fb6c405c2af83abfeb311f130707a0e219175677e366cc45d1' locally
docker.io/kindest/node@sha256:f52781bc0d7a19fb6c405c2af83abfeb311f130707a0e219175677e366cc45d1: Pulling from kindest/node
Digest: sha256:f52781bc0d7a19fb6c405c2af83abfeb311f130707a0e219175677e366cc45d1
docker: Cannot overwrite digest sha256:f52781bc0d7a19fb6c405c2af83abfeb311f130707a0e219175677e366cc45d1.
See 'docker run --help'.

When I un-set this env var, kind create cluster worked normally. I think this tracks with @tommyknows's multi-platform digest diagnosis. (I happened to be using powershell at the time but i'm sure the same thing would happen with any other shell.)

> $env:DOCKER_DEFAULT_PLATFORM = ''
> kind create cluster --name kindcluster
Creating cluster "kindcluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kindcluster"
You can now use your cluster with:

kubectl cluster-info --context kind-kindcluster

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

BenTheElder · 2023-03-30T06:15:41Z

The kind issue mentioned in above comment (#43188 (comment)) is somewhat different, more context in kubernetes-sigs/kind#2718.

On an unrelated note I've hit this working on kind itself in a way unrelated to kubernetes-sigs/kind#2718, because if you docker run --platform=linux/arm64 image/foo:bar@sha256:abcd and then docker run --platform=linux/amd64 image/foo:bar@sha256:abcd you get:

docker: Cannot overwrite digest sha256:abcd.

where abcd is the digest of a manifest list.

The platform specific manifest workaround does work but ... it's pretty clunky

samdoran · 2024-01-18T20:25:04Z

I just ran into this today and can confirm it is related to pulling a multi-arch image by SHA. The simplest way to reproduce the error is to download a multi-arch image by SHA (which actually points to a manifest, not an image) for one architecture then another.

Example Failure

> docker pull --platform=linux/arm64 quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8: Pulling from project-koku/koku-metrics-operator
...
Digest: sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
Status: Downloaded newer image for quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8

> docker pull --platform=linux/x86_64 quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8: Pulling from project-koku/koku-metrics-operator
...
Digest: sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
Cannot overwrite digest sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8

> docker rmi 9afac0396181 87e90227e2e1

> docker pull --platform=linux/x86_64 quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8: Pulling from project-koku/koku-metrics-operator
...
Digest: sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
Status: Downloaded newer image for quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8
quay.io/project-koku/koku-metrics-operator@sha256:fa59df2b2ea01274e45dd9c207fc2b86dd1dd1ee6ce54d904ead0a63120fafb8

docwhat · 2024-02-22T19:02:21Z

Can confirm this is still and issue:

❯ docker version
Client:
 Version:           24.0.7-rd
 API version:       1.42 (downgraded from 1.43)
 Go version:        go1.20.10
 Git commit:        72ffacf
 Built:             Wed Nov  1 18:41:50 2023
 OS/Arch:           darwin/arm64
 Context:           rancher-desktop

Server:
 Engine:
  Version:          23.0.6
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.20.11
  Git commit:       9dbdbd4b6d7681bd18c897a6ba0376073c2a72ff
  Built:            Fri Nov 17 20:59:57 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.7.2
  GitCommit:        0cae528dd6cb557f7201036e9f43420650207b58
 runc:
  Version:          1.1.7
  GitCommit:        860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 docker-init:
  Version:          0.19.0
  GitCommit:

❯ docker pull --platform=linux/amd64 jruby:9@sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290
docker.io/library/jruby@sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290: Pulling from library/jruby
Digest: sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290
Status: Image is up to date for jruby@sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290
docker.io/library/jruby:9@sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290

❯ docker pull --platform=linux/arm64 jruby:9@sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290
docker.io/library/jruby@sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290: Pulling from library/jruby
Digest: sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290
Cannot overwrite digest sha256:eadaf46a86eafb86e0b5aa72b39fd69fd4339196be613c546182003a85803290

tommyknows mentioned this issue Jul 22, 2022

test: fix tests on ARM (MYC-116) snyk/snyk-docker-plugin#436

Merged

3 tasks

thaJeztah mentioned this issue Sep 26, 2022

manifest versioned with SHA256 #44144

Closed

BenTheElder mentioned this issue Mar 30, 2023

upgrade kindnetd and storage provisioner to latest images kubernetes-sigs/kind#3148

Merged

Itxaka mentioned this issue Jul 4, 2024

Pin dependencies kairos-io/kairos#2677

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-platform docker run errors with "Cannot overwrite digest" when using image manifest #43188

Multi-platform docker run errors with "Cannot overwrite digest" when using image manifest #43188

marcosnils commented Jan 25, 2022 •

edited

Loading

thaJeztah commented Feb 3, 2022

honnix commented Apr 13, 2022

tommyknows commented Jul 22, 2022

mcascone commented Dec 14, 2022

BenTheElder commented Mar 30, 2023 •

edited

Loading

samdoran commented Jan 18, 2024

docwhat commented Feb 22, 2024

Multi-platform docker run errors with "Cannot overwrite digest" when using image manifest #43188

Multi-platform docker run errors with "Cannot overwrite digest" when using image manifest #43188

Comments

marcosnils commented Jan 25, 2022 • edited Loading

thaJeztah commented Feb 3, 2022

honnix commented Apr 13, 2022

tommyknows commented Jul 22, 2022

mcascone commented Dec 14, 2022

BenTheElder commented Mar 30, 2023 • edited Loading

samdoran commented Jan 18, 2024

docwhat commented Feb 22, 2024

marcosnils commented Jan 25, 2022 •

edited

Loading

BenTheElder commented Mar 30, 2023 •

edited

Loading