coredns-arm64 docker images tagged with incorrect architecture (amd64)

[bpetrikovics]

What happened:
Cannot deploy coredns-arm64 image in an arm64 swarm (e.g. as a service/stack) because the image architecture is set to amd64. Apparently image architecture is not checked if not running in swarm mode/as service, can deploy as expected with docker-compose. However it does not work in a swarm.

When attempting, the service is not deployed and stays in pending state.

Issue can be found by checking the service state and inspecting the image:

$ docker service ls
ID             NAME                          MODE         REPLICAS   IMAGE                              PORTS
xoekln0752qb   coredns_coredns               replicated   0/1        coredns/coredns-arm64:latest       *:53->53/udp

$ docker service ps xoekln0752qb --no-trunc
ID                          NAME                IMAGE                                                                                                  NODE      DESIRED STATE   CURRENT STATE           ERROR                                                  PORTS
mxdw2z4aidl0zaeq32147eljj   coredns_coredns.1   coredns/coredns-arm64:latest@sha256:224c4ecc9d9eea3765d0beee0e624e6cf837230c370440bd38a7d9901dd04dc4             Running         Pending 3 minutes ago   "no suitable node (unsupported platform on 2 nodes)"

$ docker images | grep coredns
coredns/coredns-arm64       latest    04464ed27bb9   12 days ago    49.5MB

$ docker inspect coredns/coredns-arm64 | grep -i arch
        "Architecture": "amd64",

$  uname -m
aarch64

What you expected to happen:
Image architecture is properly set and can be deployed on amd64 hosts.

How to reproduce it (as minimally and precisely as possible):

Deploy the image on an arm64 host as service (e.g. in swarm mode).

Anything else we need to know?:

Environment:

• the version of CoreDNS: coredns/coredns-arm64 latest 04464ed27bb9 (CoreDNS-1.9.0)
• Corefile: (not relevant)
• logs, if applicable:
• OS (e.g: cat /etc/os-release): Debian GNU/Linux 11 (bullseye)
• Others: docker-ce 5:20.10.123-0debian-bullseye

Edit: fixed typo + added uname output

[yongtang]

@bpetrikovics There was an effort to fix this though the hard requirement of binfmt/qemu broke our build system so it was reverted: #4779 (comment)

[yongtang]

@bpetrikovics On a side note, are you facing the issue with coredns/coredns:latest (with manifest) as well on arm64?

[yongtang]

The manifest of coredns/coredns seems to be right:

$ docker manifest inspect coredns/coredns
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:0ea13057996255e8785ffa4750fae5efd982f2c4976e58590616f5448f580c12",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:287fbb15b6021b3b6f7d2be1cd4ed0b4a197f90f7fa74a28b615cbaeb3c0c2f7",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:224c4ecc9d9eea3765d0beee0e624e6cf837230c370440bd38a7d9901dd04dc4",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:a2d1dfb89cb11574f40eab8693446291e632d76703f6255db544cf2734474829",
         "platform": {
            "architecture": "mips64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:e57db1026905ced2d7993a02f603c11d2383529fdcdff33b5cf8ac67fa444875",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:bee80d4d88da0dd4b8d040d4b23ac16a29b9bb8222d000cf0b47e95e4e158890",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

[bpetrikovics]

Using coredns/coredns:latest works, even though it also has architecture set to amd64.

$ docker service ls | grep coredns
ID NAME MODE REPLICAS IMAGE PORTS
kvkei854o970 coredns_coredns replicated 1/1 coredns/coredns:latest *:53->53/udp

$ docker service logs -f kvkei854o970
coredns_coredns.1.mlpcnpo2kzhy@pi-node-1 | goto.:53
coredns_coredns.1.mlpcnpo2kzhy@pi-node-1 | CoreDNS-1.9.0
coredns_coredns.1.mlpcnpo2kzhy@pi-node-1 | linux/arm64, go1.17.6, ace3dcb

$ docker inspect coredns/coredns:latest | grep Arch
"Architecture": "amd64",

So at this point I'm fine, thank you. It just means that people shouldn't use the arch specific images, but rather the "plain" coredns/coredns?

[yongtang]

@bpetrikovics I think the issue is rather the confusion that arch-specific image tags are "visible". Ideally, only plain coredns/coredns should be visible and accessible with tags.

All arch-specific images should only be referenced in manifest and downloadable, not through tags.

As a comparison, as far as I can see it looks like python does not have arch-specific images tagged, only one tag with all archs:
https://hub.docker.com/_/python?tab=description&page=1&ordering=last_updated&name=2.7

Maybe we can just delete the tags of all arch-specific images so they will not be "visible".

[yongtang]

Added a PR #5201.