Supporting building multi-platform images (podman buildx)

[junaruga]

Description

Supporting building multi-platform images (podman buildx)

Detail

This ticket is a request for feature, originally from containers/podman#3063 .

docker buildx [1][2] is to enable building and running multi-platform container images.
I would like to see that podman has like the feature.

$ docker buildx build --platform linux/arm64 ...

RHEL 8 started supporting multi arch including ARM 64 bit.
Quay 3 started supporting multi arch including ARM 64-bit. [3]
So, it might be a good timing for podman to support this feature.

docker buildx is using QEMU internally to do it.
As an another way to achieve this, there is qemu-user-static [4] also using QEMU.

According to the docker buildx's article [2], maybe both have similar logic in it.

This fast and lightweight container OS comes packaged with the QEMU emulator, and comes pre-configured with binfmt_misc to run binaries of any supported architecture.

But docker buildx looks much easier than qemu-user-static.

• [1] docker buildx GitHub: https://github.com/docker/buildx#building-multi-platform-images
• [2] docker buildx article: https://engineering.docker.com/2019/04/multi-arch-images/
• [3] https://www.redhat.com/en/blog/introducing-red-hat-quay-3-registry-your-linux-and-windows-containers
• [4] qemu-user-static: https://github.com/multiarch/qemu-user-static

[junaruga]

I like to explain more context about this ticket.

Why is this feature important for me now?

First, below announcement from Docker is April, 30, 2019. Just 20 days ago.
It's happening now.

[2] docker buildx article: https://engineering.docker.com/2019/04/multi-arch-images/

docker buildx is not stable and limited release for the edge version for now.
But after it is stable at once, people recognize and notice how building a multi arch container on x86_64 is easy. Currently most of the container users might not know even it is possible to build much arch containers on x86_64.

It's something like people who do not know a smart phone in a old era.
When they know it at once, they want to use it.

And people might say "We want to build and run a multi arch container like docker buildx in the open source license world, or in Fedora, CentOS or Debian ecosystem.

multiarch project

Let me share about multiarch project too that I am working for now.

[4] qemu-user-static: https://github.com/multiarch/qemu-user-static

multiarch project https://github.com/multiarch is to give people the below multi arch experience.

$ uname -m
x86_64

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -t multiarch/fedora:30-aarch64 uname -m
aarch64
$ docker run --rm -t multiarch/fedora:30-s390x uname -m
s390x

$ docker run --rm -t multiarch/fedora:29-aarch64 uname -m
aarch64
$ docker run --rm -t multiarch/fedora:29-ppc64le uname -m
ppc64le
$ docker run --rm -t multiarch/fedora:29-s390x uname -m
s390x

It enables people to add multi arch testing cases to their current CI, and it has a role to promote the multiarch container use cases.
People recognize that "oh it is possible to build, run and test multi arch cases on the CI".

Here are the cases to add the multi arch cases to Travis CI.

https://github.com/BenLangmead/bowtie2/blob/master/.travis.yml#L22-L41
https://github.com/jts/nanopolish/blob/master/.travis.yml#L46-L75

It seems that adding multi arch cases on their existing x86_64 base CI is more casual for them rather than adding new native much arch supporting CI newly like "Shippable CI', in my experience of pull-requests on some projects.

Shippable CI supports native 64/32 bit ARM environment like this.
But I have never succeeded to merge my pull-request to the other organization's repositories.
https://github.com/lh3/minimap2/pull/398/files

There are some challenges for this multiarch project's current technology.
For example, people have to use the compatible containers for qemu-user-static rather than using their existing mutl arch containers.

And docker buildx and podman buildx solve it.

[rhatdan]

If the community wants to work on this, then we would consider it. Currently we don't have the resources or the priority to work on it.

[junaruga]

@rhatdan you mean "the community" in "the community wants to work on this" means multarch project?

[rhatdan]

Yes, we need contributors opening PR's to make this happen, it is not as high as other features in the priority list of the core maintainers.

[junaruga]

@rhatdan sure. thanks for considering keeping the possibility.
I wish this ticket is kept as opening.
Though I can not promise the work for buildash, it's not zero percent. The multiarch project needs some works before being ready for the work of buildash.

[grooverdan]

Working on at least part of this with #1368 (comment). This covers the build side of buildah build-using-dockerfile --platform linux/aarch64/variant ... Yes I'd planned some interaction with qemu-static in the build.

I see docker buildx nicely integrates multiple platform arguments to produce a manifest. I'll see what can be done for buildah. I haven't looked at pod yet.

[junaruga]

Wow, great work, @grooverdan

[rhatdan]

@grooverdan Thanks for looking into this.

[rhatdan]

@grooverdan Any progress?

[grooverdan]

Yes:
tree: https://github.com/grooverdan/buildah/tree/args-target-platform
sample output of simple test case: https://dpaste.de/obgx

Current resolving issues:

• multistage images currently don't get the right target architecture

I'm making the fetching of the last FROM image in a docker file to fetch using the --platform arguments.

For nicely handling multi-stages on cross builds I think the following is needed:
extending FROM to take a --platform {os/arch/variant|"build"|"target"} argument

Its needed as in multistage it not clear if a FROM, excluding the last one, is intended to be of the build or target arch (and if we're doing that we may as well do any).

Is this an acceptable extension?

Currently only see the platform Variant being used in the Push/Pull/Manifest protocols and not an Image metadata. I'm I missing anything in this respect?

[rhatdan]

I am fine with extending the FROM command.
@nalind WDYT?

[nalind]

As a rule, I lean away from making up our own extensions to Dockerfile instructions and syntax.
What's the use case for having FROM do something other than assuming that the image being referred to is for target platform?

[grooverdan]

When you have a multiple FROM statements in a multi-stage image its fairly obvious that the last one is of the target. Earlier stages however are more ambiguous, they could be of the build (native) architecture to cross-compile some artefacts, or build architecture agnostic artefacts (which is faster not emulated), or they could be of the target architecture to have components.

For now I'll follow the implementation of docker, leave this as a separate feature.

@nalind, seems docker buildx is already using the platform extension:
https://github.com/docker/buildx#building-multi-platform-images