Multi architecture deployment using different Dockerfiles?

[rfasouliotis]

Hello,

I am wondering if there is a "clean" method of pushing multi-arch images to DockerHub, without using a separate tag.

Essentially I am trying to achieve the magic that docker buildx build --platform linux/amd64,linux/arm64 --tag <user>/<repo>:<tag> . --push provides, (i.e. one tag with multi-arch images), but using a multi-step build process since I need to use a different Dockerfile per architecture.

To circle back to my first sentence, the only way I have been able to achieve this is as follows:

• create and push the first image: docker buildx build --platform linux/amd64 -f Dockerfile1--tag <user>/<repo>:<tag> . --push
• create and push the second image, modifying the tag name: docker buildx build --platform linux/arm64 -f Dockerfile2 --tag <user>/<repo>:<tag>_arm64 . --push
• use imagetools to append: docker buildx imagetools create --append -t <user>/<repo>:<tag> <user>/<repo>:<tag>_arm64

When I go to my Docker Hub and look in this repo, under I see the digests of the 2 arch builds, which is what I want. Of course I also see a separate tag, <tag>_arm64 which comes from the build in the second step, and as I understand it, this is necessary because you can't append an image from the local repo to a remote repo, so I have to push it to DockerHub before using imagetools.

Is there a better way of doing this that doesn't require the creation and pushing of the second tag?

[tonistiigi]

You can use push-by-digest=true output option to push image without a tag https://github.com/moby/buildkit#imageregistry and then use digest reference in imagetools create

[thaJeztah]

since I need to use a different Dockerfile per architecture.

I don't know your specific situation, but with BuildKit, you're also able to make stages conditional, based on build-args. The built-in BUILDPLATFORM, TARGETPLATFORM, BUILDOS, TARGETOS, BUILDARCH, and TARGETARCH can be used to make stages conditional, based on the native (BUILDxxx) or target (TARGETxxx) platform.

For example, the following Dockerfile has both arm64 and amd64 specific stages, as well as "common" stages, which will be run on any platform;

# syntax=docker/dockerfile:1

FROM alpine AS base

# arm64-specific stage
FROM base AS build-arm64
RUN echo "This stage is used on arm64"

# amd64-specific stage
FROM base AS build-amd64
RUN echo "This stage is used on amd64 (x86)"

# common steps
FROM build-${TARGETARCH} AS build
RUN echo "This stage is used on all architectures"

Building the above with --platform=linux/amd64 will use the build-amd64 stage:

docker build --no-cache --platform=linux/amd64 --progress=plain .
...
#9 [build-amd64 1/1] RUN echo "This stage is used on amd64 (x86)"
...
#10 [build 1/1] RUN echo "This stage is used on all architectures"
...
#10 DONE 0.4s

And building teh above with --platform=linux/arm64 will use the build-arm64 stage:

docker build --no-cache --platform=linux/arm64 --progress=plain .
...
#9 [build-arm64 1/1] RUN echo "This stage is used on arm64"
...
#10 [build 1/1] RUN echo "This stage is used on all architectures"
...
#10 DONE 0.3s

[rfasouliotis]

You can use push-by-digest=true output option to push image without a tag https://github.com/moby/buildkit#imageregistry and then use digest reference in imagetools create

Thanks for the reply @tonistiigi. I was aware of the option, but not quite sure how to use it. If I don't use a tag I get:

docker buildx build --platform linux/amd64 --output=type=image,push-by-digest=true --push  .
[+] Building 0.0s (0/0)                                                                                                                                                                    
error: tag is needed when pushing to registry

If I use the same command, but add a tag in the <user>/<repo>:<tag> format, it succeeds, however it doesn't actually seem to push anything to Docker Hub. I see something like this below:

 => exporting to image                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                               0.0s
 => => exporting manifest sha256:452a766aca296d435403f87acaf7b0xxxxxxxxxxxx                                                                                    0.0s
 => => exporting config sha256:6c36dbad8558e3e52b6fe50dc890659e4141axxxxxxxxxxxxxxxxx           

and confirm by doing:

docker buildx imagetools inspect <user>/<repo>:<tag>
INFO[0000] trying next host - response was http.StatusNotFound  host=registry-1.docker.io
error: docker.io/xxxxxxx/xxxxxxx:vTEST: not found

If I add the --push option (or ,push=true to --output) I see:

 => ERROR exporting to image                                                                                                                                                          0.0s
 => => exporting layers                                                                                                                                                               0.0s
 => => exporting manifest sha256:452a766aca296d435403f87acaf7b0cb120f4xxxxxxxxxxx                                                                                     0.0s
 => => exporting config sha256:6c36dbad8558e3e52b6fe50dc890659e4141a0fa9c6xxxxxxxx                                                                                       0.0s
------
 > exporting to image:
------
error: failed to solve: can't push tagged ref docker.io/xxxxxxx/xxxxxx:vTEST by digest

Any idea on what I am doing wrong? I wasn't able to find too much information on examples on using push-by-digest=true.

This is my buildx setup if it helps:

docker buildx ls
NAME/NODE            DRIVER/ENDPOINT             STATUS   PLATFORMS
laughing_goldberg    docker-container                     
  laughing_goldberg0 unix:///var/run/docker.sock inactive 
mybuild *            docker-container                     
  mybuild0           unix:///var/run/docker.sock running  linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default              docker                               
  default            default                     running  linux/amd64, linux/386

[rfasouliotis]

@thaJeztah thank you for that. Excellent explanation on how to use the stages. I used that method for some of our repos, but we have a couple of cases where I couldn't work around separate Dockerfiles.

[tonistiigi]

can't push tagged ref docker.io/xxxxxxx/xxxxxx:vTEST by digest

Your name needs to be only the repo name. Without the :vTEST suffix.

[rfasouliotis]

@tonistiigi thank you!