Docker: Multi-arch & cross-compile build with docker buildx

[jrajahalme]

Note that this PR adds functionality that currently already exists in images/
to the main Dockerfiles.

Add support for Docker BuildX multi-arch builds for arm64 and amd64 to
Cilium runtime, builder, and release images. Add Go cross-compilation
support to Cilium release images to make the arm64 builds faster on
amd64 hosts.

If both DOCKER_BUILDKIT=1 and DOCKER_BUILDX=1 are defined, the
makefiles create a cross-compilation builder instance ('cross') and
switch buildx to use it.

Images depend on each other, so the images normally should be built in
a specific order.

0. Export variables

$ export DOCKER_BUILDKIT=1
$ export DOCKER_BUILDX=1
$ export DOCKER_REGISTRY=docker.io
$ export DOCKER_DEV_ACCOUNT=your-account

1. Runtime

$ make docker-image-runtime
$ docker buildx imagetools inspect ${DOCKER_REGISTRY}/${DOCKER_DEV_ACCOUNT}/cilium-runtime:2020-11-30

Then update the runtime image references in other Dockerfiles

2. Builder

$ make docker-image-builder
$ docker buildx imagetools inspect ${DOCKER_REGISTRY}/${DOCKER_DEV_ACCOUNT}/cilium-builder:2020-12-01

Update the main Cilium Dockerfile with the new builder reference

3. Hubble

Hubble builds via buildx QEMU integration, unless you have an ARM machine added to your buildx builder.

• Check out hubble branch pr/jrajahalme/docker-buildx-support

$ export IMAGE_REPOSITORY=${DOCKER_REGISTRY}/${DOCKER_DEV_ACCOUNT}
$ IMAGE_TAG=v0.7.1x CONTAINER_ENGINE="docker buildx" DOCKER_FLAGS="--push --platform=linux/arm64,linux/amd64" make image

$ docker buildx imagetools inspect ${IMAGE_REPOSITORY}/hubble:v0.7.1x

Update the main Cilium Dockerfile with the new Hubble reference

4. Cilium release images

Cilium Dockerfiles are modified for cross-compilation based on the multi-arch support in the Cilium builder image:

$ make docker-images-all

[jrajahalme]

test-me-please

[jrajahalme]

test-me-please

[jrajahalme]

Tests passed with test builds of runtime and builder, now removed the last commit to not impose un-official builds.

[jrajahalme]

test-me-please

[kkourt]

This is a tricky one to review...

I wish there was a cleaner way to implement all of this, but I'm not aware of any.
It might be a good idea to have some additional eyes on this, so I'll request additional reviews for my groups.

[jrajahalme]

test-me-please

[jrajahalme]

retest-gke

[jrajahalme]

retest-4.9

[joestringer]

FWIW I've been running DOCKER_BUILDKIT=1 for a while now and it seems to work well enough for me (though it can hide legitimate useful build output and I haven't figured out how to ensure it doesn't hide that on demand yet..)

I also tested this PR in its current state with buildkit enabled (but not buildx) for both of the make dev-docker-image and make microk8s targets, and there doesn't seem to be any kind of regression.

But then I also noticed that this PR changes the Dockerfile.builder but those changes are not being consumed by the main docker image build yet because there's no change to the main dockerfile to use a newer version of the builder image that would be generated by these code changes. So I think we need to generate images based on these latest changes and then add another change on top to integrate the use of those new builder images into the main dockerfiles.

Couple of other minor nits below.

[jrajahalme]

Rebased and addressed review comments.

[jrajahalme]

test-me-please

[jrajahalme]

But then I also noticed that this PR changes the Dockerfile.builder but those changes are not being consumed by the main docker image build yet because there's no change to the main dockerfile to use a newer version of the builder image that would be generated by these code changes. So I think we need to generate images based on these latest changes and then add another change on top to integrate the use of those new builder images into the main dockerfiles.

These changes allow multi-arch builds for the builder and runtime. After this PR is merged we should make a decision to build the builder and runtime for both arm64 and amd64. I can't say if that can be easily automated yet, though, so it needs to be a separate discussion.

[jrajahalme]

test-me-please

[jrajahalme]

retest-gke

[jrajahalme]

test-me-please

[jrajahalme]

retest-gke

[jrajahalme]

retest-gke

[jrajahalme]

retest-gke