mixed architecture in opensuse tumbleweed images in versions starting from 3.0.9

[ElKiwos]

Kairos version:

NAME="openSUSE Tumbleweed"
# VERSION="20240414"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20240414"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
# CPE 2.3 format, boo#1217921
CPE_NAME="cpe:2.3:o:opensuse:tumbleweed:20240414:*:*:*:*:*:*:*"
#CPE 2.2 format
#CPE_NAME="cpe:/o:opensuse:tumbleweed:20240414"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"
KAIROS_FAMILY="opensuse"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_ID="kairos"
KAIROS_PRETTY_NAME="kairos-standard-opensuse-tumbleweed v3.0.6-v1.29.3-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/opensuse:tumbleweed-standard-arm64-rpi4-v3.0.6-k3sv1.29.3-k3s1"
KAIROS_IMAGE_LABEL="tumbleweed-standard-arm64-rpi4-v3.0.6-k3sv1.29.3-k3s1"
KAIROS_TARGETARCH="arm64"
KAIROS_VERSION="v3.0.6-v1.29.3-k3s1"
KAIROS_ID_LIKE="kairos-standard-opensuse-tumbleweed"
KAIROS_VERSION_ID="v3.0.6-v1.29.3-k3s1"
KAIROS_FLAVOR_RELEASE="tumbleweed"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_NAME="kairos-standard-opensuse-tumbleweed"
KAIROS_FLAVOR="opensuse"
KAIROS_VARIANT="standard"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_ARTIFACT="kairos-opensuse-tumbleweed-standard-arm64-rpi4-v3.0.6-k3sv1.29.3+k3s1"
KAIROS_MODEL="rpi4"
KAIROS_RELEASE="v3.0.6"
KAIROS_SOFTWARE_VERSION="v1.29.3+k3s1"

CPU architecture, OS, and Version:
Linux 72af8f086b52 6.9.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 12 Jun 2024 20:17:17 +0000 aarch64 aarch64 aarch64 GNU/Linux

Describe the bug
Upgrade from opensuse-tumbleweed-standard-arm64-rpi4-v3.0.6-k3sv1.29.3-k3s1 to opensuse-tumbleweed-standard-arm64-rpi4-v3.0.9-k3sv1.29.3-k3s1 fails with

/usr/sbin/suc-upgrade: line 15: /usr/bin/kairos-agent: cannot execute binary file: Exec format error

To Reproduce

• Have any arm64 opensuse-tumbleweed-standard deployment running with a version <= 3.0.6.
• Upgrade to a version >= 3.0.9

Expected behavior
Upgrades to versions >= 3.0.9 are possible on opensuse-tumbleweed-standard

Logs
Log output of the apply job on the first node:

Defaulted container "upgrade" out of: upgrade, drain (init)
+ HOST_DIR=/host
+ '[' '' '!=' true ']'
+ diff /etc/os-release /host/etc/os-release
+ mount --rbind /host/dev /dev
+ mount --rbind /host/run /run
+ kairos-agent upgrade --source dir:/
/usr/sbin/suc-upgrade: line 15: /usr/bin/kairos-agent: cannot execute binary file: Exec format error`

Additional context

I first discovered this phenomenon while trying to upgrade my raspberry pi cluster from opensuse-tumbleweed-standard-arm64-rpi4-v3.0.6-k3sv1.29.3-k3s1 to opensuse-tumbleweed-standard-arm64-rpi4-v3.0.9-k3sv1.29.3-k3s1.
This resulted in the error of the system-upgrade job of the first node with the following error:

Defaulted container "upgrade" out of: upgrade, drain (init)
+ HOST_DIR=/host
+ '[' '' '!=' true ']'
+ diff /etc/os-release /host/etc/os-release
+ mount --rbind /host/dev /dev
+ mount --rbind /host/run /run
+ kairos-agent upgrade --source dir:/
/usr/sbin/suc-upgrade: line 15: /usr/bin/kairos-agent: cannot execute binary file: Exec format error`

Examining the images locally my development machine (amd64)

3.0.9:

Running the 3.0.9 arm64 image with docker run -it quay.io/kairos/opensuse:tumbleweed-standard-arm64-rpi4-v3.0.9-k3sv1.29.3-k3s1@sha256:ff1bf40d1ca19a074510214c9e8aede30efaf089182542a783018e1b3a9c70a8 "/bin/sh"

sh-5.2# file $(which kairos-agent)
/usr/bin/kairos-agent: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header

sh-5.2# file /usr/bin/bash
/usr/bin/bash: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=5fe3979bcc55347779f7eeb79a2a2f5b17a18ba3, for GNU/Linux 3.7.0, stripped

3.0.6:

Running the 3.0.6 arm64 image with docker run -it quay.io/kairos/opensuse:tumbleweed-standard-arm64-rpi4-v3.0.6-k3sv1.29.3-k3s1@sha256:f725f6cf5a8fd07a141695a416ad918c923606a35cd9fd5fc74f340f14e1e75e "/bin/sh"

sh-5.2# file $(which kairos-agent)
/usr/bin/kairos-agent: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, no section header
sh-5.2# file /usr/bin/bash
/usr/bin/bash: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=5fe3979bcc55347779f7eeb79a2a2f5b17a18ba3, for GNU/Linux 3.7.0, stripped

[jimmykarily]

This happened because of human error while backporting fixes on 3.0.X. The problem won't exist in 3.1.0 onwards (because CI handles this correctly). We should definitely avoid this happening when we backport but our focus is on getting 3.1.0 stable out. We won't cut a 3.0.X patch release just for this but will make sure it's fixed if we end up cutting a 3.0.X release anyhow.

[jimmykarily]

Fix will be there in 3.1.0 (actually it works fine on 3.1.0-alpha already) : #1792

thus I'm closing this.

[ElKiwos]

Thank you very much for the explanation.

[mauromorales]

I've added also a notification on all affected releases

@ElKiwos I think v3.0.4 should not have the issue in case you prefer not to consume the alpha release

[mauromorales]

@ElKiwos we have fixed this issue on v3.0.14

[ElKiwos]

Thank you very much @mauromorales for backporting this
Quick question, I tried looking into this myself before creating the issue and I couldn't find the location where the amd64 binaries are being pulled in the build pipeline for the arm64 images. Still with the backported fix I can't pinpoint it. Could you please point me in the direction where to look?

[mauromorales]

@ElKiwos sure thing!

The main Kairos repo, has a Dockerfile that is common to all distros, there one of the sources is Kairos' framework image

kairos/images/Dockerfile.kairos

Line 2 in b174f3d

FROM --platform="linux/${TARGETARCH}" quay.io/kairos/framework:${FRAMEWORK_VERSION} AS framework

Normally, this image, includes all the latest versions of the packages, which you can find on https://packages.kairos.io/

Normally, those images are multi-arch https://quay.io/repository/kairos/framework?tab=tags&tag=latest

However, for the backported fixes (most of the 3.0.x releases) we made the mistake to manually copy the backported packages e.g.

https://github.com/kairos-io/kairos-framework/blob/b769ae53a2cd843540b13b8d5a9a8ff26a31e25a/Dockerfile#L9

Which breaks arm because the arm packages live on a different repo https://quay.io/repository/kairos/packages-arm64

So for the latest release all I did is to add targets per targetarch

https://github.com/kairos-io/kairos-framework/blob/821d5f44efb5b09760bed3cd11bea0db09b7d872/Dockerfile#L10-L13

But long term, we should instead find a way to backport that plays well with the traditional way of using the multi-arch images.

I hope I did a good job explaining it, but please don't hesitate to ask any questions because this process is not trivial.