[v20.10.5 ] Platform architecture detection

[mvadu]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Since 20.10.5 (with fix for #1170 platform detection is broken other way on SBCs running 64bit kernel and 32bit user land. Raspberry Pi 4 in my case.

Expected behavior

docker pull should use the highest available platform. Trying to pull an image with arm64 support should work on a 64 bit kernel engine.

Actual behavior

docker fails to detect the arm64 part, and tries to pull arm/v7, which is not supported by image.

Steps to reproduce the behavior

1. Install fresh raspi os (either get 64 bit image, or enable kernal using arm_64bit=1 in /boot/config.txt
2. install docker
3. issue docker pull influxdb

docker pull influxdb
Using default tag: latest
latest: Pulling from library/influxdb
no matching manifest for linux/arm/v7 in the manifest list entries

4. Manually setting platform to arm64 works fine

docker pull influxdb --platform=arm64
Using default tag: latest
latest: Pulling from library/influxdb
Digest: sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe
Status: Image is up to date for influxdb:latest
docker.io/library/influxdb:latest

5. Trying to add platform: arm64 to a compose file and trying to use in docker stack deploy fails with services.influxdb Additional property platform is not allowed

Output of docker version:

 docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:46 2021
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:18 2021
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

 docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 9
  Running: 4
  Paused: 0
  Stopped: 5
 Images: 10
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: mjp1ksry2ijdrvc6sbc00a1w7
  Is Manager: true
  ClusterID: 8011srp12bt8pu7fvhw60lvqo
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.23.3
  Manager Addresses:
   192.168.23.3:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.17-v8+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.806GiB
 Name: raspberrypi4
 ID: KAH2:56LY:CSA5:PWR4:LY5D:IBZR:Q2PG:2PBN:VZU4:U3FW:USBL:NXWT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

[cpuguy83]

Can you paste logs associated with the pull?

[mvadu]

Are you referring to CLI output?

I cleared all setup, fresh start - sudo service docker stop ; sudo rm -r /var/lib/docker/; sudo service docker start

>docker pull influxdb
Using default tag: latest
latest: Pulling from library/influxdb
no matching manifest for linux/arm/v7 in the manifest list entries

>docker pull influxdb --platform=arm64
Using default tag: latest
latest: Pulling from library/influxdb
c78c297fb0d0: Pull complete
06af62193c25: Pull complete
8b846e1b7390: Pull complete
7fa5b994e094: Pull complete
912bfc01b864: Pull complete
f1872ae9f82e: Pull complete
14971371c2fa: Pull complete
218ef0f3c9be: Pull complete
faa40b3dbe64: Pull complete
Digest: sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe
Status: Downloaded newer image for influxdb:latest
docker.io/library/influxdb:latest

If there is any CLI verbose logging that I can enable to show the failed pull, please let me know.

[cpuguy83]

No I mean the docker daemon logs.
journalctl -fu docker

[mvadu]

Here you go.. keep in mind, i did a service restart per my last comment.

-- Logs begin at Fri 2021-03-05 11:59:45 EST. --
Mar 05 13:03:01 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:01.851488326-05:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Mar 05 13:03:01 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:01.852219466-05:00" level=info msg="Loading containers: start."
Mar 05 13:03:02 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:02.636474039-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.062972196-05:00" level=info msg="Loading containers: done."
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.237784709-05:00" level=info msg="Docker daemon" commit=363e9a8 graphdriver(s)=overlay2 version=20.10.5
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.238439294-05:00" level=info msg="Daemon has completed initialization"
Mar 05 13:03:03 raspberrypi4 systemd[1]: Started Docker Application Container Engine.
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.616195342-05:00" level=info msg="API listen on /var/run/docker.sock"
Mar 05 13:04:36 raspberrypi4 dockerd[16137]: time="2021-03-05T13:04:36.182066506-05:00" level=info msg="Attempting next endpoint for pull after error: no matching manifest for linux/arm/v7 in the manifest list entries"
Mar 05 13:27:59 raspberrypi4 dockerd[16137]: time="2021-03-05T13:27:59.599378523-05:00" level=info msg="Attempting next endpoint for pull after error: no matching manifest for linux/arm/v7 in the manifest list entries"

[cpuguy83]

Sorry, can you add "debug": true to /etc/docker/daemon.json... from there you can either restart the daemon or sudo kill -s HUP $(pidof dockerd).

So far from what I can tell, docker is detecting your system as armv7 instead of arm64, but debug logs should help in knowing what it is doing.

[mvadu]

gotcha.. Took me some time to realize there is no config by default. I created one with just

 $ cat /etc/docker/daemon.json
{
  "debug": true
}

 $ sudo service docker stop ; sudo rm -r /var/lib/docker/; sudo service docker start

 $ docker pull influxdb
Using default tag: latest
latest: Pulling from library/influxdb
no matching manifest for linux/arm/v7 in the manifest list entries
 $ docker pull influxdb --platform=arm64
Using default tag: latest
latest: Pulling from library/influxdb
c78c297fb0d0: Pull complete
06af62193c25: Pull complete
8b846e1b7390: Pull complete
7fa5b994e094: Pull complete
912bfc01b864: Pull complete
f1872ae9f82e: Pull complete
14971371c2fa: Pull complete
218ef0f3c9be: Pull complete
faa40b3dbe64: Pull complete
Digest: sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe
Status: Downloaded newer image for influxdb:latest
docker.io/library/influxdb:latest
 $ journalctl -fu docker
-- Logs begin at Fri 2021-03-05 11:59:45 EST. --
Mar 05 15:37:48 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:48.944599273-05:00" level=debug msg="Applying tar in /var/lib/docker/overlay2/9e3ad0d2ab92d51ed7214f399d56b96ba5c13c94f750bd65ab1d37261592504a/diff" storage-driver=overlay2
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.146893428-05:00" level=debug msg="Applied tar sha256:7ef7ab563542d66d5c58c71c12d95652292d9e6e39d51a3f811a5aa53896cc2f to 9e3ad0d2ab92d51ed7214f399d56b96ba5c13c94f750bd65ab1d37261592504a, size: 0"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.342553334-05:00" level=debug msg="Using /usr/bin/unpigz to decompress"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.353787110-05:00" level=debug msg="Applying tar in /var/lib/docker/overlay2/e688297a996993147fc0cbacf9e2f096d31979f32691a0bcd527fc2c90c3373b/diff" storage-driver=overlay2
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.528390635-05:00" level=debug msg="Applied tar sha256:95da00a24c55336b4d45eb480b17eaea03b80d598f663ad4e0e9fb9c31a9000a to e688297a996993147fc0cbacf9e2f096d31979f32691a0bcd527fc2c90c3373b, size: 98"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.709004470-05:00" level=debug msg="Using /usr/bin/unpigz to decompress"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.720995702-05:00" level=debug msg="Applying tar in /var/lib/docker/overlay2/9c4eb12b8f9f3f9c66b776f18e218e4c7a208bec602b9fc1b39d76c1a36e773a/diff" storage-driver=overlay2
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.896104870-05:00" level=debug msg="Applied tar sha256:8fed90b5e4a6a62e8fdb10a8f078242cea6ad6218e667500c29a9aa39f3dd3b8 to 9c4eb12b8f9f3f9c66b776f18e218e4c7a208bec602b9fc1b39d76c1a36e773a, size: 12229"
Mar 05 15:37:50 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:50.117550091-05:00" level=debug msg="Adding content digest to lease" digest="sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe" lease="moby-image-sha256:3186e07f4050523da24af0ff8e09695e1372807260fb00e96c9d40b04daaaa2c" remote="docker.io/library/influxdb:latest"
Mar 05 15:37:50 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:50.152887477-05:00" level=debug msg="Adding content digest to lease" digest="sha256:7418b74da890e76ef4b62152b0ce8f66b5191a067dd3d337ff5ba468acfdd9bd" lease="moby-image-sha256:3186e07f4050523da24af0ff8e09695e1372807260fb00e96c9d40b04daaaa2c" remote="docker.io/library/influxdb:latest"
^C
 $

[cpuguy83]

And is that the armv7 version instead of arm64?

[mvadu]

I am pulling this https://hub.docker.com/_/influxdb/ and it does not have a arm7 image. It has an arm64 version.

[cpuguy83]

And it works on 20.10.3?

[cpuguy83]

Also, what's the output of file $(which dockerd)

[mvadu]

that is my problem.. I did an apt update/upgrade last night, and I am not sure what version I was using before it..

file $(which dockerd)
/usr/bin/dockerd: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, Go BuildID=KWfZsgtA0v8ydK4uwzH9/W4Cc15Zw3VpW_aeXUmAA/wuC7GIfnzgPe7eXgTeJ_/aCn2OG9BBROphPzG3uIw, BuildID[sha1]=5b9deff00f2b995dcdbb844bc8724e155395238d, not stripped

[cpuguy83]

What's in /etc/apt/sources.list.d/docker.list?

[mvadu]

cat /etc/apt/sources.list.d/docker.list
deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable

[mvadu]

but if I do

docker run --rm  ubuntu uname -a
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
e0e04e4d364e: Pull complete
4c7c5587c59a: Pull complete
2e4f1be617f9: Pull complete
Digest: sha256:b4f9e18267eb98998f6130342baacaeb9553f136142d40959a1b46d6401f0f2b
Status: Downloaded newer image for ubuntu:latest
Linux c507964803a1 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 aarch64 aarch64 GNU/Linux

It pulls the aaarch64 image

[cpuguy83]

I'm not sure why armhf was detected there when the list was installed.
You can change that to arm64, assuming you have arm64 architecture enabled... dpkg --add-architecture arm64... but if its not enabled already there could be a reason, I'm not familiar with Raspbian.

It pulls the aaarch64 image

That information isn't in the image, its reading from /proc/cpuinfo, which is from the host.