Support multi-architecture pipeline directly in cue

[TomChv]

Intro

With #1087, it's now possible to config one architecture when you run a pipeline.
The aim is to improve that to :

• Add another layer of configuration that targets one pipeline and not the whole plan.
• Specify architecture directly in cue
• Manage multiple architectures in a single plan.

This way, we will be able to :

• Build & push multi-architecture images and then resolve Move to dagger when multi-arch images are supported dubo-dubon-duponey/docker-images#11
• Create multi-arch CI/CD for multi-platform application

There are already two issues (#106 and #437) related to that topic but I prefer writing that one as a proposal.

Proposal

As discussed in #1071, we will need to improve our init operation to support a flag arch.

💡 I consider that an init operation is an op.#Op used at the beginning of a pipeline, for instance: #FetchContainer, #DockerBuild.

It has no sense to give an architecture constraint to #FetchHTTP or FetchGit so they are omitted.

DockerBuild

op.#DockerBuild already supports a platforms flag.

#DockerBuild: {  
 do: "docker-build"  
 // We accept either a context, a Dockerfile or both together  
 context?:        _  
 dockerfilePath?: string // path to the Dockerfile (defaults to "Dockerfile")  
 dockerfile?:     string  
  
 platforms?: [...string]  
 buildArg?: {  
 // FIXME: should be `[string]: string | #Secret` (circular import)  
 [string]: string | _ @dagger(secret)  
 } label?: [string]: string  
 target?: string  
 hosts?: [string]: string  
}

First, we will need to fix #437, here platforms is an array of platforms. It's really cool because with that flag you can build an image in multiple architectures.
But, it doesn't work on dagger because #up is single architecture.

As a fix, we should rewrite platform as platform: string and allow only 1 architecture constraint.

💡 It's not relevant to create a type #Arch that lists each architecture available because we certainly going to lose performance.
The error will be caught in dagger, not in cue (for the moment)

Then if you want to build a multi-arch image, you can simply do

images: [arch=string]: dagger.#Artifact

images: amd64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/amd64"
    }
]

images: arm64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/arm64"
    }
]

// ... Do other stuff

FetchContainer

Thanks DockerBuild we can build an image with a specific architecture, then it's should also be possible to fetch an image with a specific architecture

Below the current definition of FetchContainer

#FetchContainer: {  
 do:  "fetch-container"  
 ref: string  
}

We will be able to config the architecture by adding an optional field arch to the definition.

E.g

#FetchContainer: {  
 do:  "fetch-container"  
 ref: string
 arch?: string
}

💡 arch must be optional since by default the architecture is the linux/amd64 OR the one configured in values.yaml.

PushContainer

As well, @dubo-dubon-duponey needs to push a multi-architecture image.
To do so, it's required to improve PushContainer to handle it.

Here's below his current definition :

#PushContainer: {  
 do:  "push-container"  
 ref: string  
}

I made a simple proof of concept to push a multi-architecture image. The only required informations are the state and the image configuration.
Those pieces of information can be extracted from a dagger.#Artifact.

So, to support a multi-architecture image, we can just add an optional field that could take as parameters artifact and then we push it instead of pushing the current pipeline state.

#PushContainer: {  
 do:  "push-container"  
 ref: string
 images?: [arch=string]: dagger.#Artifact
}

💡 It's necessary to use a map instead of an array to keep dependency working well. Either, images will always be an empty array.

I don't know if it's a cue issue but it's required to make it works.

Details

The main challenge to correctly push multi-architecture images is to push the manifest list.

This way we can get something like this

docker buildx imagetools inspect docker.io/alpine:latest                                                                                  
Name:      docker.io/library/alpine:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a
           
Manifests: 
  Name:      docker.io/library/alpine:latest@sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/library/alpine:latest@sha256:18c29393a090ba5cde8a5f00926e9e419f47cfcfd206cc3f7f590e91b19adfe9
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6
             
  Name:      docker.io/library/alpine:latest@sha256:e12ff876f0075740ed3d7bdf788107ae84c1b3dd6dc98b3baea41088aba5236f
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7
             
  Name:      docker.io/library/alpine:latest@sha256:b06a5cf61b2956088722c4f1b9a6f71dfe95f0b1fe285d44195452b8a1627de7
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

To do so, we must :

• Retrieve all states & images to push
• Push each image one by one
• Push a manifest list containing metadata about images.

Our current Export function can only push a single image.

We will need to create a new function named Exports to push multiple images.

Example

Then let me show you an example of a plan that pushes a multi-architecture image

images: [arch=string]: dagger.#Artifact

// NOTE: It can be done through a for loop
// 
images: amd64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/amd64"
    }
]

images: arm64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/arm64"
    }
]

remoteImages: #up: [
	op.#PushContainer & {
		ref: "docker.io/example/example:multi-arch"
		"images": images
	}
]

Side effects

• We will improve the docker package to expose those features to the user.

Don't hesitate to ping me if you see more side effects.
I'll keep that issue updated.

Thank you, don't hesitate to share your opinion, especially about the cue integration.

/cc @aluzzardi @talentedmrjones @samalba @shykes @dubo-dubon-duponey

[dubo-dubon-duponey]

Thrilled to see good progress on this! Thanks a lot for this work!

I'm not completely sure I understand the proposal here. Specifically, I'm wondering if we are not conflating together different concepts (the host platform onto which we are building == BUILDPLATFORM in dockerfile/buildkit lingo with the target platform == TARGETPLATFORM in dockerfile/buildkit lingo).

The ability to control BUILDPLATFORM is nice to have, but can already be done (forcing an arch specific sha instead of using a multi-arch one), and most of the time you do not want that.

Whether a build container that is being spun is to be on the host architecture, or a foreign one (through qemu) is purely a buildkit decision based on what the Dockerfile is specifying through FROM --platform.

Again assuming I understand the proposal:

images: amd64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/amd64"
    }
]

images: arm64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/arm64"
    }
]

that above ^ means: "start a container on amd64, and build for amd64, then start on arm64, and build for arm64".

is not necessarily what is desired (or at least should be left to buildkitd to decide / is already specified in Dockerfiles).

For example, for the following dockerfile:

FROM --platform=$BUILDPLATFORM SOMEIMAGE as step1

ARG BUILDPLATFORM
ARG TARGETPLATFORM

FROM debian as step2

Controls which step should run on what architecture (step1 is typically used when cross-compiling go for multiple different archs - step2 is typically for assembly or when compiling "natively" - through QEMU).

Again, maybe I'm missing the point / understanding this wrong - apologies if that's the case - either way happy to chat about this if you want (next week).

[dubo-dubon-duponey]

Also:

arch must be optional since by default the architecture is the linux/amd64 OR the one configured in values.yaml

Mmm... shouldn't it default to the host architecture of what the buildkit daemon is running on instead?

[TomChv]

@dubo-dubon-duponey Thanks for your comment! 🚀

I'm not completely sure I understand the proposal here. Specifically, I'm wondering if we are not conflating together different concepts (the host platform onto which we are building == BUILDPLATFORM in dockerfile/buildkit lingo with the target platform == TARGETPLATFORM in dockerfile/buildkit lingo).

I was talking about the TARGET_PLATFORM.
The purpose is to say "ok, build me an image that works on linux/XXXX platform", so I deducted I talk about the Target.

Also:
arch must be optional since by default the architecture is the linux/amd64 OR the one configured in values.yaml
Mmm... shouldn't it default to the host architecture of what the buildkit daemon is running on instead?

I wanted to use the default host platform too as the default architecture but it led to an error for M1 users.
@samalba encountered that issue so we choose to use linux/adm64 as the default architecture. This way, nothing changes for our current user but if they want to config the architecture for the whole plan, it's possible ;)

As well, to know in which architecture buildkit daemon is running, we can list workers and retrieve their platform but they can be multiple so how should we make a choice?
I think I've already talked about this with @aluzzardi

[dubo-dubon-duponey]

I was talking about the TARGET_PLATFORM.
The purpose is to say "ok, build me an image that works on linux/XXXX platform", so I deducted I talk about the Target.

So, in your proposal:

images: amd64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/amd64"
    }
]

images: arm64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/arm64"
    }
]

I assume platform: "FOO" is controlling the target - but then, what is the use of images: "FOO": #up?
There is already everything in the FROM --platform instruction in Dockerfiles to force a builder architecture (or ask it to stick with the host one when cross-compiling).

Furthermore, platform: really must allow an array somehow (otherwise you will end-up working against buildkit in-build strategy and rebuild multiple times stages that do not need to).

For example, think about:

FROM --platform=$BUILDPLATFORM debian as step0

RUN touch /foo

FROM scratch

COPY --from=step0 /foo /foo

No matter how many target-platforms you have, step0 should only get built once per run.
With your proposal, it will get built as many times as you have architectures I believe.

I wanted to use the default host platform too as the default architecture but it led to an error for M1 users.
@samalba encountered that issue so we choose to use linux/adm64 as the default architecture. This way, nothing changes for our current user but if they want to config the architecture for the whole plan, it's possible ;)

This might not be a good idea.
If my host is X (X != amd64), without qemu, dagger will just not work by default. Forcing users to hardwire their own platform (for that host) seems both redundant and not portable.

Overall, I would posit that the expectation is that my container will run with the host platform of my buildkit by default (if fed a multi-arch image).

As well, to know in which architecture buildkit daemon is running, we can list workers and retrieve their platform but they can be multiple so how should we make a choice?

I'm not sure why you would need / want to force that explicitly. Why not let buildkitd make that decision for you?

I hope this is helpful - I just want to make sure you do not loose time going in a direction that would make dagger less functional than raw buildctl.

[aluzzardi]

Duplicate of #1371