Multiarch Docker container

[githubsaturn]

Currently the docker containers are built for various architectures, but they are tagged separately. This means that an end user needs to know their architecture tag, when they want to run docker run certbot/certbot:architecture-tag

A more appropriate way to do this is by using Docker manifest that packages multiple images under the same tag. Most widely used images are currently using this technique. See nginx for example:

---

Moving to this new behavior is quite simple using buildx. A simple build and push that used to be like this:

docker build -t myuser/myimage:tag .
docker push myuser/myimage:tag

will change to

export DOCKER_CLI_EXPERIMENTAL=enabled
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t myuser/myimage:tag --push .

It is worth noting that the buildx is quite stable. It is, in fact, powering the standard docker build command. The only reason that is part of the experimental package, AFAIK, is that Docker is not committing to keeping the same CLI signature - this should be fine for a build process. They haven't change the CLI args in the past year, even if they do, Cerbot docker needs to update the build script. No risk to the builds.

[ohemorange]

@bmw I think I saw a discussion going by about our docker builds but I wasn't following it so I'll leave this in your capable hands

[bmw]

While I'm interested in both providing a manifest and using buildx, as you said, despite them being quite stable, Docker still flags them as experimental features so I think we probably won't adopt them until they remove that designation.

[githubsaturn]

Generally it's okay to use experimental features in the release pipeline. If things break due to a version upgrade of Docker, you can always go and change the flag to match the new requirement and be done with it.

IMHO the benefit and value it provides vastly exceed the potential headache of a small modification in the build script.

Most, if not all, other "high profile" Docker images have proper multiarch images. As an alternative, you can use docker manifest command. It is not as seamless as buildx - but it'll get the job done. So here are the changes that you need to make:

1. Build and push all images exactly the way you currently do.

2. Add this script as the last step of your build process after all images are pushed to DockerHub:

docker manifest create $CERTBOT_IMAGE_NAME:$CERTBOT_VERSION \
                       $CERTBOT_IMAGE_NAME:amd64-$CERTBOT_VERSION \
                       $CERTBOT_IMAGE_NAME:arm64v8-$CERTBOT_VERSION \
                       $CERTBOT_IMAGE_NAME:arm32v6-$CERTBOT_VERSION


docker manifest push $CERTBOT_IMAGE_NAME:$CERTBOT_VERSION

EDIT: nevermind, realized that manifest is also an experimental feature. But to this point, if other high profile images are using this technique, it's probably okay for certbot to follow the same path.

[bmw]

Thanks for the follow up info. I agree there are a lot of high profile projects using these experimental features, but I still have the same hesitation for now. Maybe we'll pick this up in the future even if it's still flagged as experimental when we have more spare time.

I'll also mention that docker manifest is also an experimental feature that "should not be used in production environments". See https://docs.docker.com/engine/reference/commandline/manifest/.

[danielwagn3r]

From a usability point of view it would be desirable to have the same tags used for all supported platforms.

[ldrrp]

I came here looking to report this issue with nginx as an example. So happy to see this ticket exactly how i would have posted it. +1 this. I have a project i would like to support amd64 and raspberry pi's but i would really hate to have to make different docker-compose files or a fork just for arm32 and arm64. Would be nice for this to work like nginx container

[MC-DeltaT]

Is Docker Buildx even experimental any more? If you look at the summary page it doesn't mention anything about being experimental or to avoid using it in production or whatever. I don't think Docker is discouraging us from using it.

The thing is that multi-arch support is such an amazing feature that, in my opinion, it's a no-brainer to use it. And note that this is the opinion of many others too - if you look at basically any popular Docker image, you'll see it uses multi-arch builds. If Docker pull support for this feature because it's "experimental", then don't worry, the majority of Docker Hub is going down with you.

The problem with not having multi-arch builds is that it cripples everyone who uses your images in an application they want to be cross-platform.
For example, I want to portably run certbot in my app on Raspberry Pi, while being able to develop on an x86 Windows machine. Well I pretty much just can't, because there's no multi-arch support. And then it's a pain to use certbot any other way, because basically the other only supported method is snap... Which doesn't work in Docker (as far as I'm aware)... And isn't supported on Windows...
On the other hand, with multi-arch builds, everything works nicely by default, no messing around required.

[Roming22]

One more vote to enable multi-arch. Not just because it's more convenient for the user, but also because it simplifies the generation and reuse of deployments.yml for k8s.

[humanoid2050]

Buildx is stable. It now ships as part of Docker Desktop and Docker Engine when installed via most package managers. https://docs.docker.com/build/install-buildx/
Adding multiarch support is a lost cost/high return improvement, especially as Arm-based servers become more prevalent.

[bmw]

Nice! It does seem like buildx is no longer flagged as experimental. docker manifest still is flagged that way unfortunately though with blog posts from Docker in the last 7 months containing text like:

Note: However, you should only use the docker manifest command in testing — not production. This command is experimental. We’re continually tweaking functionality and any associated UX while making docker manifest production ready.

[humanoid2050]

Using buildx with multiarch support actually means you don't have to invoke docker manifest. So long as you have a registry to use as a destination with the --push flag, then buildx does everything for you in one shot.