"podman build" silently adds a variant (to local parent images)

[rb83]

Issue Description

While building images locally on my M2 (arm64) that are intended to run on linux/amd64 machines, I've encountered a strange phenomenon; in that (locally) built parent images were not found while attempting to build child images that depended on these.

Images were initially duly built by passing --platform=linux/amd64 to podman build. Child images however could only be built by specifying --platform=linux/amd64/v8 (note the nonexisting "v8" variant for amd64).

Interestingly enough, skopeo inspect does not reveal a variant, nor does podman inspect. Hence I assume something is lost (or rather, added) in translation, while podman build looks up the parent image. But I'm not versed enough in the inner workings of podman to interpret this conclusively.

This might be an earlier issue that has reappeared, the limited information and troubleshooting I've found did not really help.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create two containerfiles:

baseAFile:

FROM busybox
RUN pwd | tee pwd.txt

baseBFile:

FROM myprefix/base-a:latest
RUN echo works

2. Build the first image: podman build --platform=linux/amd64 -t myprefix/base-a:latest -f baseAFile

STEP 1/2: FROM busybox
STEP 2/2: RUN pwd | tee pwd.txt
/
COMMIT myprefix/base-a:latest
--> 82ea99988343
Successfully tagged localhost/myprefix/base-a:latest
82ea9998834312e87631dbcb9bcc46ad1103656de48738ae5abfddb3a696f38c

4. Attempt to build the second image: podman build --pull=never --platform=linux/amd64 -t myprefix/base-b:3 -f baseBFile

podman build --pull=never --platform=linux/amd64 -t myprefix/base-b:3 -f baseBFile
STEP 1/2: FROM myprefix/base-a:latest
Error: creating build container: myprefix/base-a:latest: image not known

By manually adjusting the platform the second image can be built:

podman build --pull=never --platform=linux/amd64/v8 -t myprefix/base-b:3 -f baseBFile
STEP 1/2: FROM myprefix/base-a:latest
STEP 2/2: RUN echo works
--> Using cache c5d7958ac4c284becbeae66d549af99643cbae5cbaddc0b054dba3a54cdb1ebb
COMMIT myprefix/base-b:3
--> c5d7958ac4c2
Successfully tagged localhost/myprefix/base-b:3
c5d7958ac4c284becbeae66d549af99643cbae5cbaddc0b054dba3a54cdb1ebb

Describe the results you received

Second image is not built because the first image cannot be found.

Describe the results you expected

Second image is built

podman info output

Client:
  APIVersion: 5.5.0
  BuildOrigin: brew
  Built: 1747162753
  BuiltTime: Tue May 13 20:59:13 2025
  GitCommit: ""
  GoVersion: go1.24.3
  Os: darwin
  OsArch: darwin/arm64
  Version: 5.5.0
host:
  arch: arm64
  buildahVersion: 1.40.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.07
    systemPercent: 0.37
    userPercent: 0.56
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "41"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.12.13-200.fc41.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1159102464
  memTotal: 2041810944
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250121.g4f2c8e7-2.fc41.aarch64
    version: |
      pasta 0^20250121.g4f2c8e7-2.fc41.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.aarch64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 5h 39m 33.00s (Approximately 0.21 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: <internal_redacted>
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: <internal_redacted>
      PullFromMirror: ""
    Prefix: docker.io
    PullFromMirror: ""
  search:
  - docker.io
  <redacted>:
    Blocked: false
    Insecure: false
    Location: <internal_redacted>
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: <redacted>
    PullFromMirror: ""
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106415992832
  graphRootUsed: 4560441344
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 27
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.5.0
  BuildOrigin: 'Copr: packit/containers-podman-26126'
  Built: 1747094400
  BuiltTime: Tue May 13 02:00:00 2025
  GitCommit: 0dbcb51477ee7ab8d3b47d30facf71fc38bb0c98
  GoVersion: go1.23.8
  Os: linux
  OsArch: linux/arm64
  Version: 5.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

[Luap99]

@nalind @flouthoc Another variant issue, not sure if this is the same as #6185

[nalind]

It looks like the base image carries a variant field in its config (confirm: skopeo inspect --override-arch=arm64 --override-os=linux --config --raw docker://busybox), and we're preserving it when we base an image on it.

I'm not able to reproduce the "image not found" error from the second build on my linux/amd64 system or a linux/arm64 VM (natively, or remote talking to its local podman service), though.

Do you have output from the second build with --log-level=debug? Maybe that'll shed some light on what's missing when I try to reproduce it here.

[rb83]

Sure, here you go (some internal repositories redacted):

podman build --log-level=debug --pull=never --platform=linux/amd64 -t myprefix/base-b:3 -f baseBFile
INFO[0000] podman filtering at log level debug
DEBU[0000] Called build.PersistentPreRunE(podman build --log-level=debug --pull=never --platform=linux/amd64 -t myprefix/base-b:3 -f baseBFile)
DEBU[0000] SSH Ident Key "<redacted>/.local/share/containers/podman/machine/machine" <redacted>
DEBU[0000] DoRequest Method: GET URI: http://d/v5.5.0/libpod/_ping
DEBU[0000] Loading registries configuration "<redacted>/.config/containers/registries.conf"
DEBU[0000] Found credentials for <redacted_reg_A> in credential helper containers-auth.json in file <redacted>/.config/containers/auth.json
DEBU[0000] Found credentials for <redacted_reg_B> in credential helper containers-auth.json in file <redacted>/.config/containers/auth.json
DEBU[0000] DoRequest Method: POST URI: http://d/v5.5.0/libpod/build
STEP 1/2: FROM myprefix/base-a:latest
Error: creating build container: myprefix/base-a:latest: image not known

DEBU[0000] Shutting down engines

Interestingly, I am not seeing a variant field when inspecting skopeo inspect --config containers-storage:busybox, neither with nor without the --override flags. On skopeo inspect --config containers-storage:localhost/myprefix/base-a however, the variant field is present, both on arm64 and amd64. Perhaps this is spuriously injected in the first build step? (build host is an M2, which should result to arm64/v8 per my understanding).

The "docker://" storage url seems to be very slow currently, so I will provide additional output from skopeo when my TZ reaches daytime again. Though it is my understanding (and intent) that base-a and base-b are not published to an external registry.

[nalind]

Hang on, I think I got thrown by the similarity in spellings between arm64 and amd64. The "v8" variant part of the host's platform information should be getting discarded when the target platform is specified, but it isn't.

[rb83]

I suspected as much :)

For planning, do I understand the PR correctly that a 1.40.x release is intended; fixing this behaviour?

[nalind]

That's my intent, yes. Pending reviews, of course.

[nalind]

We just tagged v1.40.1, for anyone who's looking for it.

[rb83]

Looking into how to get manually into my workflow, as I suppose podman will need some time until their next release incorporating that includes buildah > 1.40.

Thanks for the fix @nalind !

[Luap99]

Podman 5.5.1 with that fix included will be releases today.

[rb83]

@Luap99 thx for the heads-up, in that case I'll wait until everything has percolated, as I expect a podman release will be fairly quickly picked up by homebrew.