Architecture is marked as AMD64 in ARM and ARM64 images in dockerhub

[owbmoore]

Looks like a little mixup here.

[sterlingdragon]

I think this may be more than a basic tagging issue. When I pull these images onto a Raspberry Pi 4, they run, but any attempt to call kubectl exec (or docker exec) for anything but /speaker raises an "exec format error".

concretely:

• kubectl exec -n metallb-system speaker-xyz -- ls -> "exec format error"
• kubectl exec -n metallb-system speaker-xyz -- /speaker -> "port already bound" (which seems normal)

This leads me to believe that the correct (arm64) binary is getting packed with the wrong (amd64) underlying OS files.

Possibly related:

I think this may be causing metallb to not work right for L2 routing. When I restart the service, I get connectivity for ~ the ttl of my arp table entry on my client device and then it stops routing. The logs for the speakers all have entries for service updates and other management tasks, but never show incoming ARP requests, despite them being confirmed on the pod with tcpdump.

Edit: For posterity, the routing issue turned out to be #454

[sterlingdragon]

Rebuilding locally reproduced the issue with "exec format error" and the mis-tagging of the image.

Enabling experimental features on docker and modifying the tasks.py to call docker build with --platform "linux/{architecture}" resulted in an image that is properly tagged. It also resolves the issue with kubectl exec raising the "exec format error".

It doesn't resolve my other issue, though, so that's probably just a mis-configuration on my end. Now that I can uses exec I can keep trying to figure out why my setup is broken. I'll report back here if it turns out to be related to the OS files in the image.

[owbmoore]

Issue is still valid today. Latest build still showing wrong architecture.

https://hub.docker.com/layers/metallb/speaker/live-website-arm64/images/sha256-c8e4fbc59f8762e93fc4a2502af8c7f90940ef1b47f0470ad9810aa2759a9ae2?context=explore

Image is arm64 but OS/ARCH is linux/amd64

[johananl]

Thanks @sterlingdragon for the analysis. The following seems to be the case:

This leads me to believe that the correct (arm64) binary is getting packed with the wrong (amd64) underlying OS files.

This is very likely a bug either in our build automation code or in manifest-tool. I'm looking into this.

[t0b3]

@johananl I guess docker build always runs on amd64. Thus the base is FROM alpine:latest with OS/ARCH=linux/amd64.

metallb/tasks.py

Line 99 in 2219723

run("docker build "

a possible solution would be to build multi platform images using docker buildx build

see also https://circleci.com/blog/building-docker-images-for-multiple-os-architectures/

[owbmoore]

@johananl

a possible solution would be to build multi platform images using docker buildx build

agree. docker buildx is what I ended up doing to get the images I needed.

[LeducH]

@owbmoore and @t0b3 Could you elaborate what steps are for the workaround until this is fixed.

[tomasz-kuzemko]

@LeducH I managed to get it working by applying the following patch (on tag v0.9.6):

diff --git a/tasks.py b/tasks.py
index 2046fd44..22c89adb 100644
--- a/tasks.py
+++ b/tasks.py
@@ -95,7 +95,8 @@ def build(ctx, binaries, architectures, registry="docker.io", repo="metallb", ta
                     branch=branch),
                 env=env,
                 echo=True)
-            run("docker build "
+            run("docker buildx build "
+                "--platform linux/{arch} "
                 "-t {registry}/{repo}/{bin}:{tag}-{arch} "
                 "-f {bin}/Dockerfile build/{arch}/{bin}".format(
                     registry=registry,

and running inv build -a arm64. If you're on an older version of Docker CLI you might have to set environment variable export DOCKER_CLI_EXPERIMENTAL=enabled.

[champtar]

Can you send this as PR ? Does it work with quay.io ?

[t0b3]

make sure to use docker >= 19.03 to have --platform feature available.
see https://github.com/docker/buildx/#with-buildx-or-docker-1903 and
https://circleci.com/docs/2.0/building-docker-images/#docker-version

[morganchristiansson]

Another fix could be to not tag/untag the -$platform images? Not sure we need them once manifest is created. When running docker buildx build --platform x,y it doesn't create -$platform images and creates manifest image tag only.

Tho It's useful to run docker build platforms in separate jobs tho which means pushing -$platform images.

And docker registry doesn't have a good API for deleting tags... tho there is some docker hub specific REST API available tho. I've not tested this idea, but I believe the images would remain as untagged images, and be referenced by the manifest similar to the result of docker buildx build --platform x,y.