Supporting building multi-platform images (podman buildx) #3063

junaruga · 2019-05-03T12:45:43Z

/kind feature

Description

Supporting building multi-platform images (podman buildx)

Detail

This ticket is a request for feature.

docker buildx [1][2] is to enable building and running multi-platform container images.
I would like to see that podman has like the feature.

$ docker buildx build --platform linux/arm64 ...

RHEL 8 started supporting multi arch including ARM 64 bit.
Quay 3 started supporting multi arch including ARM 64-bit. [3]
So, it might be good timing for podman to support this feature.

docker buildx is using QEMU internally to do it.
As an another way to archive this, there is qemu-user-static [4] also using QEMU. But docker buildx looks much easier than it.

[1] docker buildx GitHub: https://github.com/docker/buildx#building-multi-platform-images
[2] docker buildx article: https://engineering.docker.com/2019/04/multi-arch-images/
[3] https://www.redhat.com/en/blog/introducing-red-hat-quay-3-registry-your-linux-and-windows-containers
[4] qemu-user-static: https://github.com/multiarch/qemu-user-static

The text was updated successfully, but these errors were encountered:

baude · 2019-05-05T14:39:23Z

I've tagged in a few of the buildah folks ... I'm wondering if this issue is more applicable to them.

My initial reaction here is mixed but I would like to hear from the buildah guys first.

junaruga · 2019-05-09T16:20:36Z

Thank you for considering this.

rhatdan · 2019-05-12T10:52:44Z

Yes please open an issue on the buildah repo, we can discuss it there.

junaruga · 2019-05-13T11:40:56Z

@rhatdan sure. I opened the ticket containers/buildah#1590 on the buildah repo.

junaruga · 2019-07-18T18:34:24Z

I just noticed different architecture's container works on my x86_64.
Am I dreaming?

Did podman start supporting building multi-platform images, right?

$ rpm -q podman
podman-1.4.2-1.fc30.x86_64

$ uname -m
x86_64

Fedora

$ podman pull arm64v8/fedora

$ podman run --rm -t arm64v8/fedora uname -m
aarch64

$ skopeo inspect --raw docker://registry.fedoraproject.org/fedora:30 | jq
$ podman pull docker://registry.fedoraproject.org/fedora@sha256:81bc8a5a9b0cc196faba64c311d19fccef5b74a774cd0cade49e8cd2e18aaa25
$ podman run --rm -t docker://registry.fedoraproject.org/fedora@sha256:81bc8a5a9b0cc196faba64c311d19fccef5b74a774cd0cade49e8cd2e18aaa25 uname -m
aarch64

CentOS

$ podman pull arm64v8/centos
$ podman run --rm -t arm64v8/centos uname -m
aarch64

RHEL

$ skopeo inspect --raw docker://registry.access.redhat.com/ubi8

$ podman pull docker://registry.access.redhat.com/ubi8@sha256:2c448442162d3a4288650ecf63dc41ad1526bc6a830d0af8df507aa4db32faad
$ podman run --rm -t docker://registry.access.redhat.com/ubi8@sha256:2c448442162d3a4288650ecf63dc41ad1526bc6a830d0af8df507aa4db32faad uname -m
aarch64

nalind · 2019-07-18T20:19:37Z

If you have the qemu-user-static package installed on Fedora, it'll register statically-built qemu emulators with the kernel's (by way of the systemd-binfmt service) to handle cross-arch emulation for Linux binaries. Is that what's going on?

junaruga · 2019-07-18T21:05:47Z

@nalind Thank you for your advice.
I had installed qemu-user-static. But I removed after that.
When I checked above commands, qemu-user-static was not installed.

Is it below part, right?

https://src.fedoraproject.org/rpms/qemu/blob/f30/f/qemu.spec#_1237

%if %{user_static}
%post user-static
/bin/systemctl --system try-restart systemd-binfmt.service &>/dev/null || : 
%postun user-static
/bin/systemctl --system try-restart systemd-binfmt.service &>/dev/null || : 
%endif

Now I checked my current status.

$ rpm -q qemu-user-static
package qemu-user-static is not installed

$ systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static; vendor preset: dis>
   Active: inactive (dead) since Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
Condition: start condition failed at Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 823 (code=exited, status=0/SUCCESS)

Then I installed qemu-user-static now again.

$ sudo dnf install qemu-user-static

$ systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static; vendor preset: dis>
   Active: inactive (dead) since Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
Condition: start condition failed at Wed 2019-06-26 16:24:50 CEST; 3 weeks 1 days ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 823 (code=exited, status=0/SUCCESS)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavail>

Is this right condition?

nalind · 2019-07-18T21:20:14Z

I'd expect the state to be Active: active (exited) if everything's set up, but the service just adds entries under /proc/sys/fs/binfmt_misc which tell the kernel how to run the binaries, and that can be done in other ways.

When I run the example image on a system with qemu-user-static installed specifying sleep 30 as the command to run, and then run ps from outside of the container, the process I see running in the container is /usr/bin/qemu-aarch64-static /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 30.

On a system where I don't have qemu-user-static installed, I get an expected standard_init_linux.go:203: exec user process caused "exec format error" error when attempting to run the same command.

junaruga · 2019-07-18T21:35:08Z

@nalind Do you know how to remove entries from /proc/sys/fs/binfmt_misc?
Maybe I added some entries on the different way in a past time. I assume that is the reason of my wrong state: Active: inactive (dead).
I want to turn back to the status before qemu-user-static was installed.

nalind · 2019-07-18T21:49:25Z

Per https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html, to unregister a handler, you can write -1 to it by using a command like echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64.

junaruga · 2019-07-18T22:22:43Z

@nalind Thank you! Now my machine works like your machine!

Below is the result.

I think if systemctl start systemd-binfmt includes the logic adding entries to /proc/sys/fs/binfmt_misc, systemctl stop systemd-binfmt should include the logic removing entries: echo -1 > /proc/sys/fs/binfmt_misc/status.

$ sudo su -
# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64
# exit
logout
$ podman run --rm -t arm64v8/fedora uname -m
standard_init_linux.go:207: exec user process caused "exec format error"

Then

$ sudo systemctl start systemd-binfmt

The status is Active: active (exited) now.

● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/usr/lib/systemd/system/systemd-binfmt.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2019-07-19 00:12:51 CEST; 44s ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 7867 ExecStart=/usr/lib/systemd/systemd-binfmt (code=exited, status=0/SUCCESS)
 Main PID: 7867 (code=exited, status=0/SUCCESS)

Jul 19 00:12:51 unused-4-164.brq.redhat.com systemd[1]: Starting Set Up Additional Binary Formats...
Jul 19 00:12:51 unused-4-164.brq.redhat.com systemd[1]: Started Set Up Additional Binary Formats.

$ podman run --rm -t arm64v8/fedora uname -m
aarch64

$ sudo systemctl stop systemd-binfmt

$ podman run --rm -t arm64v8/fedora uname -m
aarch64

$ sudo su -
# echo -1 > /proc/sys/fs/binfmt_misc/status
# exit
logout

$ podman run --rm -t arm64v8/fedora uname -m
standard_init_linux.go:207: exec user process caused "exec format error"

junaruga · 2019-07-18T23:36:55Z

@nalind I still do not understand below your situation, what you did.

When I run the example image on a system with qemu-user-static installed specifying sleep 30 as the command to run, and then run ps from outside of the container, the process I see running in the container is /usr/bin/qemu-aarch64-static /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 30.

What you did is like this?

$ cat Dockerfile 
FROM arm64v8/fedora

RUN dnf -y --allowerasing install \
  /usr/bin/qemu-aarch64-static \
  /usr/bin/coreutils \
  /usr/bin/sleep

$ sudo systemctl start systemd-binfmt

$ podman run --rm -t arm64v8/fedora uname -m
aarch64

$ podman build --rm -t sample .

$ podman run --rm -t sample /usr/bin/qemu-aarch64-static /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 30 &

$ podman ps
CONTAINER ID  IMAGE                    COMMAND               CREATED        STATUS            PORTS  NAMES
cca7cb5b5c1b  localhost/sample:latest  /usr/bin/qemu-aar...  4 seconds ago  Up 4 seconds ago         xenodochial_blackburn

~~On my Fedora 30, /usr/bin/coreutils command is in coreutils-single RPM package that has conflict with coreutils RPM package that has /usr/bin/sleep command.~~

~~https://src.fedoraproject.org/rpms/coreutils/blob/f30/f/coreutils.spec#_106~~

~~I could not reproduce it on my environment.~~

I could reproduce it. Is it right?

Below is good enough isn't it?

$ podman run --rm -t sample /usr/bin/sleep 30

$ podman ps
CONTAINER ID  IMAGE                    COMMAND            CREATED        STATUS            PORTS  NAMES
f614957242b2  localhost/sample:latest  /usr/bin/sleep 30  4 seconds ago  Up 3 seconds ago         modest_hellman

junaruga · 2019-07-21T18:00:18Z

I think if systemctl start systemd-binfmt includes the logic adding entries to /proc/sys/fs/binfmt_misc, systemctl stop systemd-binfmt should include the logic removing entries: echo -1 > /proc/sys/fs/binfmt_misc/status.

By the way, I opened the ticket at systemd project to remove registered /proc/sys/fs/binfmt_misc/qemu-*entries in the process of systemctl stop systemd-binfmt.
systemd/systemd#13129

openshift-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label May 3, 2019

baude assigned nalind, rhatdan and TomSweeneyRedHat May 5, 2019

junaruga mentioned this issue May 13, 2019

Supporting building multi-platform images (podman buildx) containers/buildah#1590

Closed

junaruga closed this as completed May 13, 2019

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 24, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supporting building multi-platform images (podman buildx) #3063

Supporting building multi-platform images (podman buildx) #3063

junaruga commented May 3, 2019 •

edited

Loading

baude commented May 5, 2019

junaruga commented May 9, 2019

rhatdan commented May 12, 2019

junaruga commented May 13, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading

nalind commented Jul 18, 2019

junaruga commented Jul 18, 2019 •

edited

Loading

nalind commented Jul 18, 2019

junaruga commented Jul 18, 2019 •

edited

Loading

nalind commented Jul 18, 2019

junaruga commented Jul 18, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading

junaruga commented Jul 21, 2019

Supporting building multi-platform images (podman buildx) #3063

Supporting building multi-platform images (podman buildx) #3063

Comments

junaruga commented May 3, 2019 • edited Loading

baude commented May 5, 2019

junaruga commented May 9, 2019

rhatdan commented May 12, 2019

junaruga commented May 13, 2019 • edited Loading

junaruga commented Jul 18, 2019 • edited Loading

Fedora

CentOS

RHEL

nalind commented Jul 18, 2019

junaruga commented Jul 18, 2019 • edited Loading

nalind commented Jul 18, 2019

junaruga commented Jul 18, 2019 • edited Loading

nalind commented Jul 18, 2019

junaruga commented Jul 18, 2019 • edited Loading

junaruga commented Jul 18, 2019 • edited Loading

junaruga commented Jul 21, 2019

junaruga commented May 3, 2019 •

edited

Loading

junaruga commented May 13, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading

junaruga commented Jul 18, 2019 •

edited

Loading