Docker 19.03.13 for arm32v7 build failing

[kereis]

For arm32v7, the current version of Docker on Docker Hub is 19.03.08. Meanwhile, Docker version 19.03.13 is available (See Jenkins).

I think it shouldn't make a huge difference to either use 19.03.08 or 19.03.13, but it's a little bit inconsistent though and it could possibly mean that builds with any other newer version may fail.

[tianon]

Indeed, the failure looks like the official binaries might be no good:

Step 6/12 : RUN set -eux; 		apkArch="$(apk --print-arch)"; 	case "$apkArch" in 		x86_64) dockerArch='x86_64' ;; 		armhf) dockerArch='armel' ;; 		armv7) dockerArch='armhf' ;; 		aarch64) dockerArch='aarch64' ;; 		*) echo >&2 "error: unsupported architecture ($apkArch)"; exit 1 ;;	esac; 		if ! wget -O docker.tgz "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/${dockerArch}/docker-${DOCKER_VERSION}.tgz"; then 		echo >&2 "error: failed to download 'docker-${DOCKER_VERSION}' from '${DOCKER_CHANNEL}' for '${dockerArch}'"; 		exit 1; 	fi; 		tar --extract 		--file docker.tgz 		--strip-components 1 		--directory /usr/local/bin/ 	; 	rm docker.tgz; 		dockerd --version; 	docker --version
 ---> Running in 2003701aeec2
+ apk --print-arch
+ apkArch=armv7
+ dockerArch=armhf
+ wget -O docker.tgz https://download.docker.com/linux/static/stable/armhf/docker-19.03.13.tgz
Connecting to download.docker.com (13.225.65.117:443)
saving to 'docker.tgz'
docker.tgz           100% |********************************| 53.7M  0:00:00 ETA
'docker.tgz' saved
+ tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/
+ rm docker.tgz
+ dockerd --version
Illegal instruction (core dumped)

😞

[djmaze]

The outdated docker version on arm32v7 also means it is currently not possible to use debian:bullseye on arm32. Any idea on how or when this can be fixed?

[tianon]

Unfortunately, this is a problem in the upstream Docker build process that's creating the arm* binaries on https://download.docker.com/linux/static/, so the fix needs to happen there. 😞

[djmaze]

Even more people seem to be affected this problem now. Any idea on who to ping to get this forward? Or can we help with this somehow?

[tianon]

@tiborvass @thaJeztah can one of you point me at the script that's building those binaries so I can help look into why they're not working? 🙏

I've been digging in https://github.com/docker/docker-ce-packaging/tree/4865d0386b312294a2a6f8dc5a0ed0bfbfff862c/static, but I'm not sure this is what's actually creating those? (It's a bit hard to track what's actually responsible here 😇)

[djmaze]

I just tried building dockerd 20.10.3 on an armv7 machine (which runs docker 19.03.14) using make setup in the moby repository. The resulting dockerd binary yields a segmentation fault when started.

I then manually installed a development environment on the armv7 host (with debian stretch & go 1.13). I then built the static binary using hack/make.sh from the repo. The resulting dockerd binary does not crash and seems to work.

It looks to me like the docker-based build process is somehow broken on armv7.

[dweomer]

@tianon could this be related to statically compiled CGO for arm with version of golang prior to 1.16? (there were a number of compiler/toolchain enhancements that seem to have addressed some recent segfault/panics we ran into with k3s on arm).

[kolaente]

Is this still broken for arm32 with docker 20.10.7?

[yosifkit]

arm32v6: still broken

+ bashbrew build docker:20.10.7
...
+ dockerd --version
Illegal instruction (core dumped)

+ bashbrew build docker:19.03.15
...
+ dockerd --version
Segmentation fault (core dumped)

arm32v7: still broken

+ bashbrew build docker:20.10.7
...
+ dockerd --version
Illegal instruction (core dumped)

+ bashbrew build docker:19.03.15
...
+ dockerd --version
Segmentation fault (core dumped)

[djmaze]

For those who are interested, I built a docker image which includes an up-to-date armhf version. (The armhf binaries are built manually on an armhf host, which is the only way I could get it working.)

It should be more or less a drop-in replacement for the official docker image and I am planning to add future versions as well.

https://github.com/djmaze/dind-image-with-armhf

[xoxys]

@yosifkit @tianon The arm builds (tested 20.10.16) seems to be working again, at least on armv7:

# uname -m
armv7l
# dockerd --version
Docker version 1.13.1, build 7d71120/1.13.1

Any chance to get new arm images on DockerHub?

[tianon]

Unfortunately, the latest binaries are still failing for me:

armv7:

Step 5/15 : RUN set -eux; 		apkArch="$(apk --print-arch)"; 	case "$apkArch" in 		'x86_64') 			url='https://download.docker.com/linux/static/stable/x86_64/docker-20.10.16.tgz'; 			;; 		'armhf') 	url='https://download.docker.com/linux/static/stable/armel/docker-20.10.16.tgz'; 			;; 		'armv7') 	url='https://download.docker.com/linux/static/stable/armhf/docker-20.10.16.tgz'; 			;; 		'aarch64') 	url='https://download.docker.com/linux/static/stable/aarch64/docker-20.10.16.tgz'; 			;; 		*) echo >&2 "error: unsupported architecture ($apkArch)"; exit 1 ;; 	esac; 		wget -O docker.tgz "$url"; 		tar --extract 		--file docker.tgz 		--strip-components 1 		--directory /usr/local/bin/ 	; 	rm docker.tgz; 		dockerd --version; 	docker --version
 ---> Running in 719af9812fe3
+ apk --print-arch
+ apkArch=armv7
+ url=https://download.docker.com/linux/static/stable/armhf/docker-20.10.16.tgz
+ wget -O docker.tgz https://download.docker.com/linux/static/stable/armhf/docker-20.10.16.tgz
Connecting to download.docker.com (143.204.157.26:443)
saving to 'docker.tgz'
docker.tgz            66% |*********************           | 38.1M  0:00:00 ETA
docker.tgz           100% |********************************| 57.4M  0:00:00 ETA
'docker.tgz' saved
+ tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/
+ rm docker.tgz
+ dockerd --version
Illegal instruction (core dumped)

armv6:

Step 5/15 : RUN set -eux; 		apkArch="$(apk --print-arch)"; 	case "$apkArch" in 		'x86_64') 			url='https://download.docker.com/linux/static/stable/x86_64/docker-20.10.16.tgz'; 			;; 		'armhf') 	url='https://download.docker.com/linux/static/stable/armel/docker-20.10.16.tgz'; 			;; 		'armv7') 	url='https://download.docker.com/linux/static/stable/armhf/docker-20.10.16.tgz'; 			;; 		'aarch64') 	url='https://download.docker.com/linux/static/stable/aarch64/docker-20.10.16.tgz'; 			;; 		*) echo >&2 "error: unsupported architecture ($apkArch)"; exit 1 ;; 	esac; 		wget -O docker.tgz "$url"; 		tar --extract 		--file docker.tgz 		--strip-components 1 		--directory /usr/local/bin/ 	; 	rm docker.tgz; 		dockerd --version; 	docker --version
 ---> Running in 0f17c3f44c23
+ apk --print-arch
+ apkArch=armhf
+ url=https://download.docker.com/linux/static/stable/armel/docker-20.10.16.tgz
+ wget -O docker.tgz https://download.docker.com/linux/static/stable/armel/docker-20.10.16.tgz
Connecting to download.docker.com (65.8.228.23:443)
saving to 'docker.tgz'
docker.tgz            44% |**************                  | 25.5M  0:00:01 ETA
docker.tgz           100% |********************************| 57.4M  0:00:00 ETA
'docker.tgz' saved
+ tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/
+ rm docker.tgz
+ dockerd --version
Illegal instruction (core dumped)

[xoxys]

Hmm dont get, it it works for me on a native raspberry pi...

[tianon]

I tried it on both a Raspberry Pi 1 and a Raspberry Pi 2 and they both gave me the same behavior as above. 😞

[xoxys]

Uhm now Im completely confused....

[xoxys]

Ok I have tested it on a Pi 4, but still wondering why it works there:

[~] # wget -O docker.tgz https://download.docker.com/linux/static/stable/armhf/docker-20.10.16.tgz
--2022-05-24 21:11:30--  https://download.docker.com/linux/static/stable/armhf/docker-20.10.16.tgz
Resolving download.docker.com (download.docker.com)... 108.138.7.33, 108.138.7.88, 108.138.7.48, ...
Connecting to download.docker.com (download.docker.com)|108.138.7.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 60262562 (57M) [application/x-tar]
Saving to: ‘docker.tgz’

100%[====================================================================================================================================================================================================================>] 60,262,562  9.86MB/s   in 5.9s   

2022-05-24 21:11:36 (9.76 MB/s) - ‘docker.tgz’ saved [60262562/60262562]

[~/test] # tar --extract --file docker.tgz --strip-components 1 --directory .
[~/test] # dockerd --version
Docker version 1.13.1, build 7d71120/1.13.1
[~/test] # cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (AltArch)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (AltArch)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[~/test] # uname -a
Linux computron 5.4.179-v7l.1.el7 #1 SMP Thu Feb 17 16:07:38 UTC 2022 armv7l armv7l armv7l GNU/Linux
[~/test] # uname -m
armv7l
[~/test] # cat /sys/firmware/devicetree/base/model
Raspberry Pi 4 Model B Rev 1.2

[tianon]

[~/test] # dockerd --version
Docker version 1.13.1, build 7d71120/1.13.1

I think you meant ./dockerd --version here? 😅

[xoxys]

Ouch.... can we please delete this conversation? 🙈 My bad and sorry that I have bothered you.... You are totally right, and I executed the system's dockerd 🤦 sorry again. So... that at least explains why it was working only for me, It's still not working Illegal instruction as you said.

[tianon]

Not a problem -- it was worth checking again. 👍

[Letme]

Any chance we could get this going? Now building a container on RPI Kubernetes cluster is not possible.