Add a target for building kernel with buildx #3792

fredericdalleau · 2022-06-21T16:25:36Z

- What I did
this pr contains various improvements

Add a target that uses buildx to generate a multiarch images. buildx enables remote builders to build multiarch images.
Upgrade the kernel used in the sample linuxkit image
Parallelize compression of the kernel source archive. That saves several minutes.

- How I did it
Add a new target in the makefile

- How to verify it
ORG=yourorg IMAGE=yourimage PLATFORMS=linux/arm64,linux/amd64 make -C kernel buildx_5.10.x

- Description for the changelog
kernel: Add a target for building kernel with buildx

- A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

This surpringly saves a lot of time: M1: from 340 to 90 seconds Intel: from 527 to 222 seconds (2 cores 4 threads) Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

buildx can use remote builders and automatically generate the multiarch manifest. A properly configured builder is required : First create docker context for the remote builders : $ docker context create node-<arch> --docker "host=ssh://<user>@<host>" Then create a buildx configuration using the remote builders: $ docker buildx create --name kernel_builder --platform linux/amd64 $ docker buildx create --name kernel_builder --node node-arm64 --platform linux/arm64 --append $ docker buildx use kernel_builder $ docker buildx ls Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

deitch · 2022-06-21T16:42:38Z

How does this align with the work done in linuxkit pkg build? We do lots of under-the-covers work for building multi-arch using buildx there.

deitch · 2022-06-21T16:42:50Z

@rn particularly will have to comment on this.

fredericdalleau · 2022-06-21T16:49:18Z

I used to build kernels directly using make, and had a hardtime with multiarch images, and buildx just fitted so well here. Maybe I was just doing it wrong.

deitch · 2022-06-21T16:55:43Z

Not necessarily. At the moment, we still build kernels mostly directly, not via linuxkit pkg build. So this very well may be desired. Let's let @rn weigh in.

fredericdalleau · 2022-06-23T12:25:20Z

kernel/Makefile

+			--build-arg DEBUG=$(4) \
+			--build-arg BUILD_IMAGE=$(IMAGE_BUILDER) \
+			$(LABELS) \
+			--no-cache -t $(ORG)/$(IMAGE):$(1)$(3)$(4)-$(TAG)$(SUFFIX) .


the suffix is not needed in the manifest image name

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

To update configuration for 5.10 kernels use : make -C kernel KERNEL_VERSIONS=5.10.104 kconfigx Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

rn · 2022-07-17T10:02:17Z

It would be nice to build kernels with linuxkit pkg build but the builds are a bit different. I think most of what was described in #2583 still holds

djs55

I think building kernels with linuxkit pkg build is still a good end goal to aim for. As far as I can see this PR is a small incremental improvement to the current process: builds are faster and support remote builders. I don't think this blocks or conflicts with any work on linuxkit pkg build and gives us some short-term benefits.

I think we should take this and still work on improving linuxkit pkg build.

tonistiigi · 2023-05-04T16:56:40Z

kernel/Dockerfile.kconfigx

+# Apply patches to all kernels and move config files into place
+RUN <<EOF
+set -e
+for VERSION in ${KERNEL_VERSIONS}; do


Looks like this actually only accepts one version in here because same arg is also used in COPY.

tonistiigi · 2023-05-04T16:59:53Z

kernel/Dockerfile.kconfigx

+    patch \
+    xz
+
+COPY / /


This should be after source download/extraction iiuc.

To minimize cache misses you could only pull in the config-x/patches file that you actually need. Or because you only need the context files for a single RUN you could just mount in the files with --mount=target=. for that step and avoid copy completely.

The makefile sets --no-cache actually...

Do you know why? It is also weird that it is doing a pull based on name, instead of for example doing a --cache-from from the previous image that would verify that cache is up-to-date and only pull if there is an actual cache match.

Do you know why?

I think that's just been a LinuxKit policy since the early days, see e.g. edeaf1c.

It is also weird that it is doing a pull based on name, instead of for example doing a --cache-from from the previous image that would verify that cache is up-to-date and only pull if there is an actual cache match.

If you are referring to docker pull $tag || docker build --tag $tag pattern, I am pretty sure the idea is to avoid building if a tag is already present. It's not about cache, is about whether the unique image version was already released or still needs building and releasing.

tonistiigi · 2023-05-04T17:08:03Z

kernel/Dockerfile.kconfigx

+ARG KERNEL_VERSIONS
+ARG TARGETARCH
+WORKDIR /
+COPY --from=kernel-build /linux-${KERNEL_VERSIONS}/.config config-${KERNEL_VERSIONS}-$TARGETARCH


because $TARGETARCH is not the same format and needs a special rename in a makefile, a better pattern in there could be to do something like cp .config /out/config-real-name in the RUN step and then in here just do COPY --from=kernel-build /out /

fredericdalleau added 5 commits June 21, 2022 17:53

Use latest kernel in linuxkit

4e43876

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

Parallelize kernel source compression

a117bbc

This surpringly saves a lot of time: M1: from 340 to 90 seconds Intel: from 527 to 222 seconds (2 cores 4 threads) Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

Add a PLATFORMS variable to declare platforms needed for buildx

13ab3be

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

Make image name customizable

1ab5258

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

fredericdalleau changed the title ~~Add support for building with Buildx support~~ Add a target for building kernel with buildx Jun 21, 2022

fredericdalleau commented Jun 23, 2022

View reviewed changes

fredericdalleau added 2 commits June 23, 2022 16:24

Do not tag use the architecture suffix for images built with buildx

caa0346

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

Add make kconfigx to upgrade configs using buildx

369ca6f

To update configuration for 5.10 kernels use : make -C kernel KERNEL_VERSIONS=5.10.104 kconfigx Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>

djs55 approved these changes Apr 27, 2023

View reviewed changes

deitch approved these changes Apr 28, 2023

View reviewed changes

deitch merged commit c2df261 into linuxkit:master Apr 28, 2023

tonistiigi reviewed May 4, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a target for building kernel with buildx #3792

Add a target for building kernel with buildx #3792

fredericdalleau commented Jun 21, 2022 •

edited

deitch commented Jun 21, 2022

deitch commented Jun 21, 2022

fredericdalleau commented Jun 21, 2022 •

edited

deitch commented Jun 21, 2022

fredericdalleau Jun 23, 2022 •

edited

rn commented Jul 17, 2022

djs55 left a comment

tonistiigi May 4, 2023

tonistiigi May 4, 2023

errordeveloper May 10, 2023

tonistiigi May 10, 2023

errordeveloper May 10, 2023 •

edited

tonistiigi May 4, 2023

Add a target for building kernel with buildx #3792

Add a target for building kernel with buildx #3792

Conversation

fredericdalleau commented Jun 21, 2022 • edited

deitch commented Jun 21, 2022

deitch commented Jun 21, 2022

fredericdalleau commented Jun 21, 2022 • edited

deitch commented Jun 21, 2022

fredericdalleau Jun 23, 2022 • edited

Choose a reason for hiding this comment

rn commented Jul 17, 2022

djs55 left a comment

Choose a reason for hiding this comment

tonistiigi May 4, 2023

Choose a reason for hiding this comment

tonistiigi May 4, 2023

Choose a reason for hiding this comment

errordeveloper May 10, 2023

Choose a reason for hiding this comment

tonistiigi May 10, 2023

Choose a reason for hiding this comment

errordeveloper May 10, 2023 • edited

Choose a reason for hiding this comment

tonistiigi May 4, 2023

Choose a reason for hiding this comment

fredericdalleau commented Jun 21, 2022 •

edited

fredericdalleau commented Jun 21, 2022 •

edited

fredericdalleau Jun 23, 2022 •

edited

errordeveloper May 10, 2023 •

edited