Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
discussion: builder future: buildkit #32925
I'm creating a new issue here about the buildkit proposal discussed in #32550 (comment). Although the implementation for this wouldn't be part of this repo I don't know a better place to discuss this atm. It's part of the Moby effort to eventually break up the monolith and unblock innovation on build use cases.
(originally from https://gist.github.com/tonistiigi/059fc72c4630f066d94dafb5e0e70dc6)
Buildkit is a proposal to separate out
One of the main design goals of buildkit is to separate frontend and backend concerns during a build process. A frontend is something designed for the users to describe their build definition. Backend solves the problem of finding a most efficient way to solve a common low-level description of the build operations, that has been prepared for them by the frontends.
The purpose of buildkit is not to be an arbitrary task runner. Instead, buildkit solves the problem of converting source code to an artifact in a self-contained, portable, reproducible, and most efficient way. Invoking builder should be traceable to immutable sources and invoking it shouldn't have any side-effects. Buildkit will support intelligent caching of artifacts of previous invocations so it can be efficiently used in a developer workflow.
Buildkit is meant to be used as a long-running service. It is optimized for parallel execution of complex projects and building multiple projects at the same time.
Buildkit is separated to following subcomponents:
Connection with docker platform
Buildkit is meant to become the next generation backend implementation for
When invoked from the Docker CLI buildkit would be capable of exposing clients context directory as a source and use Docker containers as a worker. The snapshots would be backed by Docker's layer store(containerD snapshot drivers). End results from the builder would be exported to
A frontend is a component that takes in user-provided build definition, parses it and prepares a generalized definition for the low-level builder.
Buildkit supports multiple frontends. Most common example of a frontend is Dockerfile. Frontend also has access to the other components of the builder. It can access the sources directly and has access to store/get resources from cache. For example, for Dockerfile to correctly parse the
The core part of the builder is a solver that takes a DAG of low-level build instructions from the frontend and finds a way to execute them in a most efficient manner while keeping the cache for the next invocations.
For this, the graph of build instructions should be loaded into a content addressable store. Evey item in that store can have dependencies from previous items. That makes sure that no definitions are duplicated. To start a builder a root node from that graph is asked to be solved with a provided worker options. That internally will call the same action for its dependencies and so on.
While solving an instruction a cache key is computed to see if a result for the instruction can be already found without computing the step. If it is found, a snapshot associated with the cache-key can be used as a result directly. After every instruction, the result of the operation is stored by the same cache key for future use.
The goal is to:
Supported operations for LLB:
LLB is optimized for simplicity. The main operation that it support is running a process in the context of one snapshot and capturing the modifications this process made. To simplify and optimize implementation there is a built-in operation for copying data from one snapshot to another and accessing data from one of the remote sources known to the builder.
Low-level builder only works on snapshots. There are no methods for controlling image metadata changes. Image metadata can be managed by a frontend, should it be needed. The only component that could know about the image format is image exporter. The
ExecOp can depend on multiple snapshots as its inputs, one of them would be mounted at
Another operation to support invoking other builders in a build operation to support nested invocation of builders. This is covered more in the ControlAPI section.
Sources is a component that allows registering transport methods to the builder that prepare remote data to the snapshot. A build operation can refer to a remote data by an identifier, that identifier is used to find the registered source provider that has the actual implementation.
Supported built-in sources include docker images, git repositories, http archives and local directories. It is likely that a source implementation uses cache from previous invocations to speed up getting access to the data. Docker image source would skip pulling image layers that it has pulled before, git source could reuse previously pulled repo and only pull in incremental changes.
When integrated with
Worker is a component tasked with running a build step. Only required steps are moving data between snapshots and executing a command with correct data mounts.
Usually worker would run a container to execute the process but that is not a requirement set by the builder.
An exporter is a post-step that runs before data is returned. Unlike
That means that the build result may be a plugin or OCI-image bundle or maybe just a file or a directory.
Snapshots cover the implementation for the filesystem modifications needed during the build. It supports plugging in different backends. When buildkit is used as part of
Snapshots API uses reference counting because the same data may be used outside the build as well. When a part of the system(build operation) takes a reference to a snapshot it can't be deleted by anything else.
The persistent storage used by buildkit is managed automatically by a cache manager component. The user can specify a simple cache policy that is used by the garbage collector to clean up unneeded resources.
The build cache contains snapshots previously accessed by the builder and some metadata for the operations(cache keys) referring to these snapshots. If a builder has stopped using a snapshot, before releasing it, it would call
The user also has control to see what is currently tracked by yhr cache manager and manually prune its contents.
Build cache can be exported out of buildkit and imported in another machine. It can be stored in the registry using a distribution manifest.
ExportCache would export a config object with metadata how the snapshots are referred by operation cache-keys. That data could be pushed to a registry with every snapshot being pushed as a separate blob. Cache importer can read back this config and expose the operation cache to the currently running builder action. If a cache-key requested by an operation is not found locally but exist in the imported configuration, snapshot associated with it can be pulled in from the registry.
Control API is an API layer for controlling the builder while it is running as a long-running service. It supports invoking a build job and inspecting how a build job would execute. The user should be able to query a build target by not executing it and get to see the vertex graph of all operations that would be executed and if they are already backed by the cache.
By defining a common interface a client program can be used with multiple builder implementations. This also enables supporting nested builder invocations as a build operation. That would be similar to
This is something that interests me, thanks for the ping.
Btw. I think this will change? github.com/docker/docker/builder -> github.com/moby/moby/builder
@alexellis This encourages new community projects that want to either provide a special interface for declaring a build definition(with frontends) or define custom low-level build functionality(with nested invocation). A project like Dockramp that reimplements
Yeah, things are in flux atm. Buildkit would be an open Moby project,
It might be good to consider Build Context as well.
Op would have a dependency on arbitrary number of contexts, and can be executed when these deps are ready.
Could we also explicitly define process/service boundaries ? for ex, should a frontend be implemented in a client CLI ? as an independent swarm service independently scalable / upgradable / pluggable ? Do we intend to use Swarm at all for that ? Now that Docker is clearly multi-platform, should we go with a multi-worker architecture so that a single Buildkit deployment can build images for Linux, Windows, ARM, whatever comes next from a single endpoint ?
Anyway, I am very interested in it :)
Buildkit should support multiple instances of workers(design tbd). For
This should be possible with multiple workers. How this is managed by user should be controlled by frontend.
referenced this issue
Jul 3, 2017
referenced this issue
Jul 17, 2017
referenced this issue
Jul 25, 2017
Bazel is also similar but
It has built-in distributed builds and that's what powers https://hydra.nixos.org and other related build environments, unless I'm misunderstanding what you mean.
This has been controversial but fully isolated builds are one flag away (and much of the community wants that flag to default to true, even if it doesn't do so today), using Linux namespaces, seccomp, and no capabilities. This means zero network connectivity inside a build, and the only parts of the filesystem visible to you are the ones you need for your dependencies. On macOS there's a different mechanism with similar properties.
Can't argue there
Anyway, I mostly just like to see acknowledgments of related work when new work is started/proposed. It's very different to know the landscape and deliberately choose to try something different, vs. accidentally reinventing the wheel. It sounds like you know what's going on but it would be lovely to get a more fleshed out version of these points in a permanent location, because I'm pretty sure I won't be the only person with these questions when they see your project.
Hey so I was wondering since this is being worked on to be moved upstream I think I can help with testing. I am sure @crosbymichael and others remember what happened the last time someone rewrote the builder and we had to test it and fix a ton of bugs. Anyways I successfully ran buildkit on all my images but I am sure mine all kinda have the same semantic structure so if you want help testing this across many I would be happy to help.
Also @tonistiigi thanks for buildkit it is truly awesome :)