Supporting building multi-platform images (podman buildx)

[junaruga]

/kind feature

Description

Supporting building multi-platform images (podman buildx)

Detail

This ticket is a request for feature.

docker buildx [1][2] is to enable building and running multi-platform container images.
I would like to see that podman has like the feature.

$ docker buildx build --platform linux/arm64 ...

RHEL 8 started supporting multi arch including ARM 64 bit.
Quay 3 started supporting multi arch including ARM 64-bit. [3]
So, it might be good timing for podman to support this feature.

docker buildx is using QEMU internally to do it.
As an another way to archive this, there is qemu-user-static [4] also using QEMU. But docker buildx looks much easier than it.

• [1] docker buildx GitHub: https://github.com/docker/buildx#building-multi-platform-images
• [2] docker buildx article: https://engineering.docker.com/2019/04/multi-arch-images/
• [3] https://www.redhat.com/en/blog/introducing-red-hat-quay-3-registry-your-linux-and-windows-containers
• [4] qemu-user-static: https://github.com/multiarch/qemu-user-static

[baude]

I've tagged in a few of the buildah folks ... I'm wondering if this issue is more applicable to them.

My initial reaction here is mixed but I would like to hear from the buildah guys first.

[junaruga]

Thank you for considering this.

[rhatdan]

Yes please open an issue on the buildah repo, we can discuss it there.

[junaruga]

@rhatdan sure. I opened the ticket containers/buildah#1590 on the buildah repo.

[junaruga]

I just noticed different architecture's container works on my x86_64.
Am I dreaming?

Did podman start supporting building multi-platform images, right?

$ rpm -q podman
podman-1.4.2-1.fc30.x86_64

$ uname -m
x86_64

Fedora

$ podman pull arm64v8/fedora

$ podman run --rm -t arm64v8/fedora uname -m
aarch64

$ skopeo inspect --raw docker://registry.fedoraproject.org/fedora:30 | jq
$ podman pull docker://registry.fedoraproject.org/fedora@sha256:81bc8a5a9b0cc196faba64c311d19fccef5b74a774cd0cade49e8cd2e18aaa25
$ podman run --rm -t docker://registry.fedoraproject.org/fedora@sha256:81bc8a5a9b0cc196faba64c311d19fccef5b74a774cd0cade49e8cd2e18aaa25 uname -m
aarch64

CentOS

$ podman pull arm64v8/centos
$ podman run --rm -t arm64v8/centos uname -m
aarch64

RHEL

$ skopeo inspect --raw docker://registry.access.redhat.com/ubi8

$ podman pull docker://registry.access.redhat.com/ubi8@sha256:2c448442162d3a4288650ecf63dc41ad1526bc6a830d0af8df507aa4db32faad
$ podman run --rm -t docker://registry.access.redhat.com/ubi8@sha256:2c448442162d3a4288650ecf63dc41ad1526bc6a830d0af8df507aa4db32faad uname -m
aarch64

[nalind]

If you have the qemu-user-static package installed on Fedora, it'll register statically-built qemu emulators with the kernel's (by way of the systemd-binfmt service) to handle cross-arch emulation for Linux binaries. Is that what's going on?

[junaruga]

@nalind Thank you for your advice.
I had installed qemu-user-static. But I removed after that.
When I checked above commands, qemu-user-static was not installed.

Is it below part, right?

https://src.fedoraproject.org/rpms/qemu/blob/f30/f/qemu.spec#_1237

%if %{user_static}
%post user-static
/bin/systemctl --system try-restart systemd-binfmt.service &>/dev/null || : 
%postun user-static
/bin/systemctl --system try-restart systemd-binfmt.service &>/dev/null || : 
%endif

Now I checked my current status.

$ rpm -q qemu-user-static
package qemu-user-static is not installed

$ systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static; vendor preset: dis>
   Active: inactive (dead) since Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
Condition: start condition failed at Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 823 (code=exited, status=0/SUCCESS)

Then I installed qemu-user-static now again.

$ sudo dnf install qemu-user-static

$ systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static; vendor preset: dis>
   Active: inactive (dead) since Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
Condition: start condition failed at Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 823 (code=exited, status=0/SUCCESS)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavail>

Is this right condition?

[nalind]

I'd expect the state to be Active: active (exited) if everything's set up, but the service just adds entries under /proc/sys/fs/binfmt_misc which tell the kernel how to run the binaries, and that can be done in other ways.

When I run the example image on a system with qemu-user-static installed specifying sleep 30 as the command to run, and then run ps from outside of the container, the process I see running in the container is /usr/bin/qemu-aarch64-static /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 30.

On a system where I don't have qemu-user-static installed, I get an expected standard_init_linux.go:203: exec user process caused "exec format error" error when attempting to run the same command.

[junaruga]

@nalind Do you know how to remove entries from /proc/sys/fs/binfmt_misc?
Maybe I added some entries on the different way in a past time. I assume that is the reason of my wrong state: Active: inactive (dead).
I want to turn back to the status before qemu-user-static was installed.