Add instructions to build multi-arch container image #2019

linki · 2022-06-16T12:40:19Z

This adds build steps to build a new multi-arch container image for skipper (and tools) for linux/amd64 and linux/arm64.

The heart of this change is the additional Makefile step called docker.push.multiarch which builds a multi-arch image using buildx by providing multiple platform arguments.

In order to do that I needed to change several parts of the Makefile but tried to keep everything backwards compatible. I would prefer to make the multi-arch build the single container image artifact and remove the separate Dockerfiles for .arm and .arm64. However, currently we don't have a public registry that supports multi-arch images where we publish skipper to, so I leave them in place for now.

The new multi-arch image is pushed to our private registry for now and since its base image only supports amd64 and arm64 at the moment we don't add the arm/v7 variant because we don't need that at Zalando.

The final multi-arch image will live at container-registry.zalando.net/teapot/skipper.

There are some small changes though:

The official image for skipper (registry.opensource.zalan.do/teapot/skipper) used to be build by the task that is now called docker.build. This task builds binaries in the local architecture into the current working directory and uses docker build to stick them into the Dockerfile. This is basically the upper half of the Makefile. However, the lower half of the Makefile consists of several more tasks to build binaries for all of the supported architectures. The official docker image is now build from these Makefile steps as well as using buildx. The resulting binaries and images are the same and one can think of removing the upper half of the Makefile now.
All the binaries in the build process are now build by the tasks in the lower half of the Makefile (build.linux.amd64 etc.). This made it very easy to build all the binaries for all architectures and then putting them all into a single multi-arch image. (Well, actually just amd64 and arm64 but you get the idea.)
Therefore, the folder structure of the output build/ folder has changed to be more consistent with the --platform variable of docker and looks like this now:

[packaging] $ tree build
build
├── darwin
│   ├── amd64
│   │   ├── eskip
│   │   ├── routesrv
│   │   ├── skipper
│   │   └── webhook
│   └── arm64
│       ├── eskip
│       ├── routesrv
│       ├── skipper
│       └── webhook
├── linux
│   ├── amd64
│   │   ├── eskip
│   │   ├── routesrv
│   │   ├── skipper
│   │   └── webhook
│   ├── arm
│   │   └── v7
│   │       ├── eskip
│   │       ├── routesrv
│   │       ├── skipper
│   │       └── webhook
│   └── arm64
│       ├── eskip
│       ├── routesrv
│       ├── skipper
│       └── webhook
└── windows
    └── amd64
        ├── eskip
        ├── routesrv
        ├── skipper
        └── webhook

10 directories, 24 files

I also added build steps for darwin/arm64 for the new M1/2 macbooks.
The naming of the *.tar.gz build artifacts have been slightly changed to be more consistent. They also contain all the binaries and not just skipper with this change.

If you want to check that all the binaries are build correctly, you can run make build.package.

Test images from a previous successful run can be found here:

registry.opensource.zalan.do/teapot/skipper-test:pr-2019-7 (single-arch amd64)
container-registry-test.zalando.net/teapot/skipper-test:pr-2019-7 (multi-arch)

…tools Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>

linki · 2022-06-16T15:08:08Z

Makefile

@@ -38,7 +38,7 @@ endif

 build: $(SOURCES) lib skipper eskip webhook routesrv

-build.linux.armv8:
+build.linux.arm64:


This change just aligns the names better with the rest of the internet:

When using arm (32-bit) then you can chose whether it's version 5, 6 or 7 (GOARM). Hence the naming is arm/v7, arm/v6 etc. When using a 64-bit arm there's no support for GOARM (see goreleaser/goreleaser#36 (comment)), hence it's just arm64 (no variant).

linki · 2022-06-16T15:09:40Z

Makefile

@@ -47,7 +47,10 @@ build.linux.armv7:
 build.linux:
 	GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o bin/skipper -ldflags "-X main.version=$(VERSION) -X main.commit=$(COMMIT_HASH)" ./cmd/skipper

-build.osx:
+build.darwin.arm64:


renamed osx to darwin to match GOOS and docker's --platform variable.

linki · 2022-06-16T15:10:14Z

delivery.yaml

      fi
-      export IMAGE ARM_IMAGE ARM64_IMAGE
+      export IMAGE ARM_IMAGE ARM64_IMAGE MULTIARCH_IMAGE


I think we can collapse this into a single image at some point. But for now I would just do it like this.

linki · 2022-06-16T15:10:36Z

delivery.yaml

+
+      cd packaging
+      make docker.build.amd64 && git status && git diff && make docker.push.amd64
+      make docker.push.multiarch


This builds and pushes a test image.

The use of && seems arbitrary - should we maybe && all of the commands or have each on the single line for clarity?

I guess it means:

do not fail on cd packaging because directory exists

each image push should be tested separately

Maybe it makes more sense to add && \ on the first 2 such that we have the same behavior as before and don't try to push multiarch if we got a compile error before.

I think besides this here everything looks fine

I separated the cd packaging from this particular command because all subsequent commands expect to be run within that folder. That was hard to spot when it's lumped together with the first build.

I'm going with && \ all commands then because that's what you both agree on. But I wonder: putting each command on its own line should have the same affect since the pipeline will stop when one command fails.

Ok, I just added && make docker.push.multiarch. Let me know if that's not fine for you.

linki · 2022-06-16T15:11:38Z

delivery.yaml

      if [[ $CDP_TARGET_BRANCH == master && ! $CDP_PULL_REQUEST_NUMBER ]]; then
        echo "Created docker image registry.opensource.zalan.do/teapot/skipper:${RELEASE_VERSION}"
+        cdp-promote-image "${MULTIARCH_IMAGE}"


This promotes the test image to the production registry. Note, this would be a no-op if we're on a branch but keeping it inside the if makes running it locally easier (since it's not executed).

linki · 2022-06-16T15:12:41Z

packaging/Dockerfile

@@ -1,8 +1,14 @@
-FROM registry.opensource.zalan.do/library/alpine-3.13:latest
+ARG BASE_IMAGE=registry.opensource.zalan.do/library/alpine-3.13:latest


Let's keep the public single-arch approved base image as default for now since this project is open source.

linki · 2022-06-16T15:17:09Z

packaging/Dockerfile

+ADD ${BUILD_FOLDER}/${TARGETPLATFORM}/skipper \
+    ${BUILD_FOLDER}/${TARGETPLATFORM}/eskip \
+    ${BUILD_FOLDER}/${TARGETPLATFORM}/webhook \
+    ${BUILD_FOLDER}/${TARGETPLATFORM}/routesrv /usr/bin/


This is not perfectly backwards compatible because of BUILD_FOLDER=build instead of BUILD_FOLDER=.. However, the corresponding task in the Makefile provides that.

Changing the default to . would require the majority of docker build invocations to overwrite it. Having the default like this only requires one task to overwrite it.

linki · 2022-06-16T15:18:04Z

packaging/Dockerfile.arm64

@@ -1,10 +1,10 @@
-FROM --platform=linux/amd64 alpine:3.13 AS build
+FROM --platform=linux/arm64 alpine:3.13


--platform is not strictly required but it ensures that we use the correct architecture for the added arm binaries below.

linki · 2022-06-16T15:18:30Z

packaging/Dockerfile.armv7

@@ -1,10 +1,10 @@
-FROM --platform=linux/amd64 alpine:3.13 AS build
+FROM --platform=linux/arm/v7 alpine:3.13


--platform is not strictly required but it ensures that we use the correct architecture for the added arm binaries below.

linki · 2022-06-16T15:19:25Z

packaging/Makefile

@@ -52,6 +56,17 @@ docker.push.arm64: docker.build.arm64
 docker.push.armv7: docker.build.armv7
 	docker push $(ARM_IMAGE)

+docker.push.multiarch: clean build.linux docker.build.enable
+	# Install qemu interpreter for arm64 (https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+	docker run --rm --privileged container-registry.zalando.net/teapot/tonistiigi-binfmt:qemu-v6.2.0-main-2 --install arm64


Please see https://github.bus.zalan.do/automata/issues/issues/4491 for why this is required and for any progress which could make this unnecessary.

linki · 2022-06-16T15:20:32Z

packaging/Makefile

+	docker run --rm --privileged container-registry.zalando.net/teapot/tonistiigi-binfmt:qemu-v6.2.0-main-2 --install arm64
+
+	# create a Buildkit builder with CDP specific configuration (https://cloud.docs.zalando.net/howtos/cdp-multiarch/)
+	docker buildx create --config /etc/cdp-buildkitd.toml --driver-opt network=host --bootstrap --use


That's a buildx builder that can build for both amd64 and arm64 and works with Zalando's internal compliance and security mechanisms on its proprietary CICD system.

linki · 2022-06-16T15:21:37Z

packaging/Makefile

+
+	# build multi-arch container image using a trusted multi-arch base image
+	docker buildx build --rm -t $(MULTIARCH_IMAGE) --platform linux/amd64,linux/arm64 --push \
+	  --build-arg BASE_IMAGE=container-registry.zalando.net/library/alpine-3.13:latest .


Here we use a proper multi-arch base image that's approved for production. Since it's currently not public it's not the default yet and only used from here.

Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>

linki · 2022-06-16T15:40:08Z

Note, that I needed to use a third party image (mirrored to our registry) in order to initialize proper arm64 emulation (without it, it doesn't work). It's the very same approach that GitHub uses for GitHub Actions so it's not bad. However, there's active discussion on what causes the issues and what's the best approach going forward here: https://github.bus.zalan.do/automata/issues/issues/4491.

For now, I hope we can also make progress with the current approach. Since it only affects the private multi-arch image we can easily track and control where it's used.

packaging/Dockerfile.arm64

packaging/Makefile

AlexanderYastrebov · 2022-06-16T15:54:27Z

packaging/Makefile

-	rm -rf skipper eskip webhook routesrv build/
+	rm -rf $(BINARIES) build/
+
+docker.build: clean $(BINARIES)


How does this relate to docker-build (with a dash) target?

It came from this version:

skipper/packaging/Makefile

Lines 40 to 45 in 41b621e

docker.build.amd64: clean skipper eskip webhook routesrv

docker build -t $(IMAGE) .

docker.build.arm64: clean build.linux.arm64 docker.build.enable

docker buildx build -t $(ARM64_IMAGE) --platform linux/arm64 -f Dockerfile.arm64 .

docker.build.armv7: clean build.linux.armv7 docker.build.enable

docker buildx build -t $(ARM_IMAGE) --platform linux/arm/v7 -f Dockerfile.arm .

which is a bit inconsistent (see usage of build and buildx)

With this: docker.build is the "local" build (no architecture suffix, builds for local architecture) and uses the steps defined before.

Whereas docker.build.amd64 etc. cross-compile for the target architectures using the steps at the bottom and buildx. docker-build and docker-push are wrappers to to invoke all three builds for the linux OS.

We could rename docker.build to docker.build.local or similar to avoid confusion.

Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>

szuecs · 2022-06-17T08:50:35Z

👍

AlexanderYastrebov · 2022-06-17T08:57:13Z

👍

add instructions to build multi-arch container image for skipper and …

c2db36f

…tools Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>

linki force-pushed the multiarch-image branch 2 times, most recently from 87d8fa8 to 762db0e Compare June 16, 2022 14:36

linki commented Jun 16, 2022

View reviewed changes

add custom multi-arch builder for CDP pipeline

f1ab89c

Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>

linki force-pushed the multiarch-image branch from 762db0e to f1ab89c Compare June 16, 2022 15:22

AlexanderYastrebov reviewed Jun 16, 2022

View reviewed changes

packaging/Dockerfile.arm64 Show resolved Hide resolved

AlexanderYastrebov reviewed Jun 16, 2022

View reviewed changes

packaging/Makefile Show resolved Hide resolved

AlexanderYastrebov reviewed Jun 16, 2022

View reviewed changes

linki force-pushed the multiarch-image branch from d87d8bf to 7400a36 Compare June 16, 2022 16:44

inline multi-arch build with current build commands

42d9dd3

Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>

linki force-pushed the multiarch-image branch from 7400a36 to 42d9dd3 Compare June 16, 2022 16:48

AlexanderYastrebov merged commit 8670e7d into master Jun 17, 2022

AlexanderYastrebov deleted the multiarch-image branch June 17, 2022 08:57

AlexanderYastrebov mentioned this pull request Sep 1, 2023

packaging: refactor docker image build #2566

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add instructions to build multi-arch container image #2019

Add instructions to build multi-arch container image #2019

linki commented Jun 16, 2022 •

edited

linki Jun 16, 2022 •

edited

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

AlexanderYastrebov Jun 16, 2022

szuecs Jun 16, 2022

szuecs Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022 •

edited

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

linki Jun 16, 2022

linki commented Jun 16, 2022 •

edited

AlexanderYastrebov Jun 16, 2022

linki Jun 16, 2022

szuecs commented Jun 17, 2022

AlexanderYastrebov commented Jun 17, 2022

		@@ -1,8 +1,14 @@
		FROM registry.opensource.zalan.do/library/alpine-3.13:latest
		ARG BASE_IMAGE=registry.opensource.zalan.do/library/alpine-3.13:latest

		@@ -1,10 +1,10 @@
		FROM --platform=linux/amd64 alpine:3.13 AS build
		FROM --platform=linux/arm64 alpine:3.13

		@@ -1,10 +1,10 @@
		FROM --platform=linux/amd64 alpine:3.13 AS build
		FROM --platform=linux/arm/v7 alpine:3.13

	docker.build.amd64: clean skipper eskip webhook routesrv
	docker build -t $(IMAGE) .
	docker.build.arm64: clean build.linux.arm64 docker.build.enable
	docker buildx build -t $(ARM64_IMAGE) --platform linux/arm64 -f Dockerfile.arm64 .
	docker.build.armv7: clean build.linux.armv7 docker.build.enable
	docker buildx build -t $(ARM_IMAGE) --platform linux/arm/v7 -f Dockerfile.arm .

Add instructions to build multi-arch container image #2019

Add instructions to build multi-arch container image #2019

Conversation

linki commented Jun 16, 2022 • edited

linki Jun 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linki Jun 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linki commented Jun 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szuecs commented Jun 17, 2022

AlexanderYastrebov commented Jun 17, 2022

linki commented Jun 16, 2022 •

edited

linki Jun 16, 2022 •

edited

linki Jun 16, 2022 •

edited

linki commented Jun 16, 2022 •

edited