Buildah in podman - Wrong architecture for base image in multistage build

[pini-gh]

Description

I've successfully set up a gitlab runner after this documentation.

It works great for single-stage Dockerfile, but when building multistage for a foreign architecture it pulls the wrong arch image at the final stage.

Note: single stage builds are OK.

Steps to reproduce the issue:

My box runs Debian testing amd64. The current podman version is 4.3.1. The package qemu-user-static is installed.

1. Set up a new buildah-issue directory with this Dockerfile:

   FROM --platform=$BUILDPLATFORM debian:bullseye-slim as build
   ARG BUILDPLATFORM
   ARG TARGETPLATFORM
   
   RUN apt update && DEBIAN_FRONTEND=noninteractive apt -y install jq dpkg-dev
   RUN mkdir -p /build && cp /usr/bin/jq /build/
   RUN dpkg-architecture
   
   # =========================================
   FROM debian:bullseye-slim
   
   RUN apt update && DEBIAN_FRONTEND=noninteractive apt -y install dpkg-dev file
   RUN dpkg-architecture
   
   COPY --from=build /build /usr/local/bin/
   RUN file /usr/local/bin/jq /bin/bash
   

   This tow stages Dockerfile is written after this Docker documentation:

   1. First stage = cross compilation
   2. second stage = installation

2. Run quay.io/buildah/stable with this folder mounted

   $ podman run --privileged --rm -it -v ./buildah-issue:/buildah-issue quay.io/buildah/stable
   

3. Enter the mounted volume and build with --layers --platform linux/amd64

   # cd /buildah-issue
   # buildah bud --layers --platform linux/amd64 --tag test:amd64-latest -f Dockerfile .
   

4. Do the same with --layers --platform linux/arm64

   # buildah bud --layers --platform linux/arm64 --tag test:arm64-latest -f Dockerfile .
   

Describe the results you received:

When building for linux/arm64 at the second stage every layer is pulled from the cache after the linux/amd64 build, and the resulting image is the very same as the linux/amd64 one:

# buildah bud --layers --platform linux/arm64 --tag test:arm64-latest -f Dockerfile .
...
[2/2] STEP 1/5: FROM debian:bullseye-slim
[2/2] STEP 2/5: RUN apt update && DEBIAN_FRONTEND=noninteractive apt -y install dpkg-dev file
--> Using cache c899df24e2ee6902c318dcbf287ccaed14b680b537b661ed0090cb4fc2a6d7ae
--> c899df24e2e
[2/2] STEP 3/5: RUN dpkg-architecture
--> Using cache bb290c9d3b3180c9e70f7ef15152d75a61a315394e6be63ab8daa81a2d448a82
--> bb290c9d3b3
[2/2] STEP 4/5: COPY --from=build /build /usr/local/bin/
--> Using cache a3e13a020d9ec7a74965eb12f92b782c05af6852e4e189f86295b436b514a121
--> a3e13a020d9
[2/2] STEP 5/5: RUN file /usr/local/bin/jq /bin/bash
--> Using cache 2032e897a756841406ca9c74aceaf09281563de44cb740dfed9b5db1579f5adc
[2/2] COMMIT test:arm64-latest
--> 2032e897a75
[Warning] one or more build args were not consumed: [TARGETARCH TARGETOS]
Successfully tagged localhost/test:arm64-latest
Successfully tagged localhost/test:amd64-latest
2032e897a756841406ca9c74aceaf09281563de44cb740dfed9b5db1579f5adc

Describe the results you expected:

The second stage base image should have pulled the linux/arm64 debian:bullseye-slim image from docker.io registry instead of pulling the linux/amd64 one from the cache.

Output of buildah version:

As said above I use buildah from image quay.io/buildah/stable. In this context:

# buildah version
Version:         1.29.1
Go Version:      go1.19.5
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.24.1
Git Commit:      
Built:           Fri Feb 17 10:05:41 2023
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of podman version on the host

$ podman --version
podman version 4.3.1
gitlab-runner@maison:~$ podman version
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of cat /etc/*release:

$ cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Output of uname -a:

$ uname -a
Linux maison 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1 (2023-03-19) x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

No such file. I don't think this is relevant in this context anyway.

[pini-gh]

As a workaround, tweaking the Dockerfile to pull the foreign arch first does work:

FROM debian:bullseye-slim as target

FROM --platform=$BUILDPLATFORM debian:bullseye-slim as build
# Do cross compilation

FROM target
# Do installation

Then I guess that the issue is that --platform information is lost after the first stage.

[flouthoc]

Hi @pini-gh , The issue happens because buildah's cache mechanism only considers history not the original architecture requested, I guess this can be fixed at buildah end, thanks for creating the issue and reporting :) I'll check this.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc any update?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.