Figure out multi-platform configuration

[sipsma]

A few questions to be answered:

1. Can/How do you supply a non-default platform for action execution?
2. Can/How do you supply multiple platforms for action execution?

[sipsma]

Just a thought, whenever I start considering how to support this I keep coming back to the idea that it could be part of a more general concept of what I'm thinking of as "ambient configuration" (don't love the term, but using it until I think of something better).

Other configuration that might fall in this bucket besides multi-platform config includes cache configuration, network configuration (including proxy setups), maybe even CA certs available to containers (companies often need to install their own).

This "ambient" type of configuration has the following properties:

1. It is applicable to potentially any extension execution (i.e. it's not specialized to one API)
2. Extension authors don't know what particular value of configuration is desired; it's only known by the end-user invoking the extension (either directly or through a transitive dependency)
   • E.g. an extension that outputs Filesystems may be able to output both x86 and ARM versions of the FS. The extension author probably isn't going to declare that it will *always *output both platform though; it's up to the end user invoking the extension to decide which platform they want to build for (or to build both).
3. It may often (but not always) be desired for the configuration of one extension to be passed (or inherited by) extensions that it invokes (e.g. I am outputting an ARM filesystem, so the extension I'm invoking to build that filesystem better output an ARM filesystem).

My idea is that we probably don't want to do something like add graphql resolver args to every single field that allows you configure which platform it should use and/or output (though nothing is certain at this point). The configuration is thus ambient in the sense that it is declared by the user when they start executing their DAG without the need to necessarily explicitly pass it through every step in the DAG.

None of this is a requirement for implementing multi-platform config, I just want to make sure that we think about whether our solution here could possibly overlap with our solution to other problems in the same bucket like cache configuration.

[sipsma]

Dumping some notes and thoughts on this

---

How BuildKit (and Dockerfile) handle multiplatform config:

• Every llb op is associated with a platform. When using the client/llb library this defaults to (in order of priority)
  1. the explicitly configured platform for the op
  2. the platform of the previous result the op is layered on top of (if any)
  3. the platform set as the marshal default
  4. the platform of the client.
• This platform has different effects for different op types:
  • Image Source - pulls the image of that platform
  • ExecOp - If the platform matches the buildkit host platform, just executes it normally. Otherwise, it executes it using qemu emulation (if setup and available on the buildkit host).
• In Dockerfiles, other information such as TARGETPLATFORM that can be used for cross-compilation (faster than emulation) is available in each step but has to be specifically used and provided to user applications running in the step.
  • This tonistiigi/xx repo has helpers that intend to make this more seamless, but still something that operates in "userspace" rather than enforced by Buildkit itself
• Running with multiple values of platform (i.e. --platforms when using buildx) results in submitting multiple solves where the marshal default platform has been set. The returned refs are then combined into a single result (using buildkit terminology) along with some platform metadata. If that result is exported as an image, buildkit will push a multiplatform image w/ a manifest list.
  • Example of combining platform specific results here
• Good reference:
  • https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

---

In the context of Cloak, a few important questions:

1. What can be configured with specific (non-default) platforms? Just the same things as LLB (e.g. core exec ops and images)? Or that plus arbitrary extensions?
2. If extensions can be configured with a given platform:
   1. What is the default value they execute as?
   2. How can that default value be overridden?
   3. Does platform configuration "ambiently" pass from an extension to any other extension it calls?
   4. Do we also support the equivalent of $TARGETPLATFORM? If so, how do extensions get access to that value at runtime?

Some starting scenarios to think through:

1. Calling only core apis from a script, I want to build a binary for N different platforms and export an image containing that binary with a manifest list that contains entries for each of the platforms
2. Same scenario as above, but the whole build is abstracted behind an extension. The script still wants to export the same multiplatform image but without any direct core api calls. The extension handles the core api calls (and thus needs to be passed platform info)
3. Same scenario as above, but now the extension doesn't even use core api calls, it directly creates platform specific files within its own execution
   • This scenario relies on features that haven't been implemented yet (but will most likely exist in the future), so it will be hard to experiment concretely with at the moment. However, it's worth at least thinking through how/whether this would work (which will inform more design decisions down the road)

For each of those scenarios, it's also worth thinking about other types of exports, e.g. if I wanted to do a local export instead of an image one (export a binary for each platform I'm building for), does it still work?

All the above scenarios just assume you use qemu emulation whenever there is a platform not matching the host platform. This means it really only handles cpu arch differences, not OS. A follow up would be to investigate how extensions could selectively use cross-compilation instead (e.g. something that gives equivalent functionality as $TARGETPLATFORM), which would be useful both for faster cross-arch builds but also builds for non-Linux OSes like macos and windows.

[sipsma]

Multiplatform support has been merged: #3119