ARMv6 machine pulls v7 image from manifest list

[max469]

Description

If I pull a multi-platform image e.g. httpd:2.1.34 with a Raspberry Pi Zero (ARMv6 CPU) it pulls the v7 image it can't execute, even though there is a v5 image in the manifest list.

I've made a few manifest lists with different variant fields in a private registry and found that the Pi with v6 and one with v7 always pulled the same images.

Here are the results of my tests. The first colum shows the variant field of the first image in the manifest list, the first row shows the variant field of the second image (nV for a missing variant field).
The values show, which image was actually pulled (always the same on both ARMv6 and ARMv7).

I1/I2	nV	v5	v6	v7
nV	-	nV	nV	nV
v5	nV	-	v5	v7
v6	nV	v6	-	v7
v7	v7	v7	v7	-

I think this result is a bit strange, because it seems like the order of the manifest list makes a difference. But that might be an other issue.

Steps to reproduce the issue:

1. install raspbian on Raspberry Pi zero
2. install docker via curl -sSL get.docker.com | sh
3. run docker run httpd:2.1.34
   It pulls the v7 Image instead of v5 and fails to execute it

Additional information you deem important (e.g. issue happens only occasionally):

Manifest list of httpd:2.1.34:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1780,
         "digest": "sha256:a2eeb4eece5b956cbd6fae9ec2d5f9fd083a197ae6f1ed68ccb01bb034893c65",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1780,
         "digest": "sha256:3413e136310f58123b51bab337d191c2412d2d630a7e7cf282bb0a92c1a53937",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v5"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1780,
         "digest": "sha256:341ab2c49183a4766c9d8d263a2920fcadc5178d16328ec1a3939d998bcc8687",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1780,
         "digest": "sha256:c1aac1569f6a3e4ea6c29ae8f5740dd466eeb894c258d5aa9797b44549081175",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      }
   ]
}

Output of docker version Raspberry Pi Zero:

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:19:46 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:15:34 2018
  OS/Arch:          linux/arm
  Experimental:     false

Output of docker info Raspberry Pi Zero:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.52+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv6l
CPUs: 1
Total Memory: 433.5MiB
Name: raspberrypi
ID: BJDS:E36I:V6AG:N4WQ:QXAU:NMTB:5BJV:UPUL:ALGC:R37E:E2ZJ:6JCV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpuset support

[thaJeztah]

Related discussion #34875

[nunofgs]

Does anyone have a workaround?

[kremerol]

@nunofgs : a workaround for 'official images' (see https://github.com/docker-library/official-images#architectures-other-than-amd64) like httpd is to prefix the image name with the 'architecture id'. In this case, "arm32v5". So, "docker run -it --rm httpd:2.4.43 uname -m" will fail, but "docker run -it --rm arm32v5/httpd:2.4.43 uname -m" will return "armv6l" on a raspberry pi one.

[tianon]

This was fixed in containerd/containerd#4530, which got pulled in here in #41688, and was officially released in v20.10.0. 👍