From ff245333ab8aece051b938dc4039f8ec69a4bc06 Mon Sep 17 00:00:00 2001 From: CrazyMax Date: Mon, 24 Oct 2022 14:30:10 +0200 Subject: [PATCH] build: extending buildkit section Signed-off-by: CrazyMax --- _data/toc.yaml | 6 ++ build/extend/dockerfile-frontend.md | 99 +++++++++++++++++++++++++++++ build/extend/index.md | 64 +++++++++++++++++++ build/images/buildkit-dag.svg | 74 +++++++++++++++++++++ build/index.md | 34 +++++----- 5 files changed, 258 insertions(+), 19 deletions(-) create mode 100644 build/extend/dockerfile-frontend.md create mode 100644 build/extend/index.md create mode 100644 build/images/buildkit-dag.svg diff --git a/_data/toc.yaml b/_data/toc.yaml index d8f4db2e9ccf..2d64ac290389 100644 --- a/_data/toc.yaml +++ b/_data/toc.yaml @@ -1541,6 +1541,12 @@ manuals: title: Build contexts and linking targets - path: /build/customize/bake/compose-file/ title: Building from Compose file + - sectiontitle: Extending BuildKit + section: + - path: /build/extend/ + title: Overview + - path: /build/extend/dockerfile-frontend/ + title: Custom Dockerfile syntax - sectiontitle: Buildx section: - path: /build/buildx/install/ diff --git a/build/extend/dockerfile-frontend.md b/build/extend/dockerfile-frontend.md new file mode 100644 index 000000000000..3cb09fd4c752 --- /dev/null +++ b/build/extend/dockerfile-frontend.md @@ -0,0 +1,99 @@ +--- +title: Custom Dockerfile syntax +keywords: build, buildkit, dockerfile, frontend +--- + +## Dockerfile frontend + +BuildKit supports loading frontends dynamically from container images. To use +an external Dockerfile frontend, the first line of your [Dockerfile](../../engine/reference/builder.md) +needs to set the [`syntax` directive](../../engine/reference/builder.md#syntax) +pointing to the specific image you want to use: + +```dockerfile +# syntax=[remote image reference] +``` + +For example: + +```dockerfile +# syntax=docker/dockerfile:1 +# syntax=docker.io/docker/dockerfile:1 +# syntax=example.com/user/repo:tag@sha256:abcdef... +``` + +This defines the location of the Dockerfile syntax that is used to build the +Dockerfile. The BuildKit backend allows to seamlessly use external +implementations that are distributed as Docker images and execute inside a +container sandbox environment. + +Custom Dockerfile implementations allows you to: + +- Automatically get bugfixes without updating the Docker daemon +- Make sure all users are using the same implementation to build your Dockerfile +- Use the latest features without updating the Docker daemon +- Try out new features or third-party features before they are integrated in the Docker daemon +- Use [alternative build definitions, or create your own](https://github.com/moby/buildkit#exploring-llb){:target="_blank" rel="noopener" class="_"} + +> **Note** +> +> BuildKit also ships with Dockerfile frontend builtin, but it is recommended +> to use an external image to make sure that all users use the same version on +> the builder and to pick up bugfixes automatically without waiting for a new +> version of BuildKit or Docker engine. + +## Official releases + +Docker distributes official versions of the images that can be used for building +Dockerfiles under `docker/dockerfile` repository on Docker Hub. There are two +channels where new images are released: `stable` and `labs`. + +Stable channel follows [semantic versioning](https://semver.org){:target="_blank" rel="noopener" class="_"}. +For example: + +- `docker/dockerfile:1` - kept updated with the latest `1.x.x` minor _and_ patch + release. +- `docker/dockerfile:1.2` - kept updated with the latest `1.2.x` patch release, + and stops receiving updates once version `1.3.0` is released. +- `docker/dockerfile:1.2.1` - immutable: never updated. + +We recommend using `docker/dockerfile:1`, which always points to the latest +stable release of the version 1 syntax, and receives both "minor" and "patch" +updates for the version 1 release cycle. BuildKit automatically checks for +updates of the syntax when performing a build, making sure you are using the +most current version. + +If a specific version is used, such as `1.2` or `1.2.1`, the Dockerfile needs +to be updated manually to continue receiving bugfixes and new features. Old +versions of the Dockerfile remain compatible with the new versions of the +builder. + +### labs channel + +The "labs" channel provides early access to Dockerfile features that are not yet +available in the stable channel. Labs channel images are released in conjunction +with the stable releases, and follow the same versioning with the `-labs` suffix, +for example: + +- `docker/dockerfile:labs` - latest release on labs channel. +- `docker/dockerfile:1-labs` - same as `dockerfile:1` in the stable channel, + with labs features enabled. +- `docker/dockerfile:1.2-labs` - same as `dockerfile:1.2` in the stable + channel, with labs features enabled. +- `docker/dockerfile:1.2.1-labs` - immutable: never updated. Same as + `dockerfile:1.2.1` in the stable channel, with labs features enabled. + +Choose a channel that best fits your needs; if you want to benefit from +new features, use the labs channel. Images in the labs channel provide a superset +of the features in the stable channel; note that `stable` features in the labs +channel images follow [semantic versioning](https://semver.org){:target="_blank" rel="noopener" class="_"}, +but "labs" features do not, and newer releases may not be backwards compatible, +so it is recommended to use an immutable full version variant. + +## Other resources + +For documentation on "labs" features, master builds, and nightly feature +releases, refer to the description in [the BuildKit source repository on GitHub](https://github.com/moby/buildkit/blob/master/README.md){:target="_blank" rel="noopener" class="_"}. +For a full list of available images, visit the [`docker/dockerfile` repository on Docker Hub](https://hub.docker.com/r/docker/dockerfile){:target="_blank" rel="noopener" class="_"}, +and the [`docker/dockerfile-upstream` repository on Docker Hub](https://hub.docker.com/r/docker/dockerfile-upstream){:target="_blank" rel="noopener" class="_"} +for development builds. diff --git a/build/extend/index.md b/build/extend/index.md new file mode 100644 index 000000000000..1c223c6cb0fc --- /dev/null +++ b/build/extend/index.md @@ -0,0 +1,64 @@ +--- +title: Extending BuildKit +description: Introduction and overview of BuildKit +keywords: build, buildkit +--- + +## Overview + +[BuildKit](https://github.com/moby/buildkit){:target="_blank" rel="noopener" class="_"} +is the backend evolution from the Legacy Builder, it comes with new and much +improved functionality that can be powerful tools for improving your builds' +performance or reusability of your Dockerfiles, and it also introduces +support for complex scenarios: + +- Detect and skip executing unused build stages +- Parallelize building independent build stages +- Incrementally transfer only the changed files in your build context between builds +- Detect and skip transferring unused files in your build context +- Use [Dockerfile frontend](dockerfile-frontend.md) implementations with many new features +- Avoid side effects with rest of the API (intermediate images and containers) +- Prioritize your build cache for automatic pruning + +Apart from many new features, the main areas BuildKit improves on the current +experience are performance, storage management, and extensibility. From the +performance side, a significant update is a new fully concurrent build graph +solver. It can run build steps in parallel when possible and optimize out +commands that don't have an impact on the final result. We have also optimized +the access to the local source files. By tracking only the updates made to these +files between repeated build invocations, there is no need to wait for local +files to be read or uploaded before the work can begin. + +## LLB + +At the core of BuildKit is a new low-level build definition format called [LLB](https://github.com/moby/buildkit#exploring-llb){:target="_blank" rel="noopener" class="_"} +(low-level builder). This is an intermediate binary format that end users are +not exposed to but allows to easily build on top of BuildKit. LLB defines a +content-addressable dependency graph that can be used to put together very +complex build definitions. It also supports features not exposed in Dockerfiles, +like direct data mounting and nested invocation. + +![Directed acyclic graph (DAG)](../images/buildkit-dag.svg){:class="invertible" style="width:80%"} + +Everything about execution and caching of your builds is defined only in LLB. +The caching model is entirely rewritten compared to the current Dockerfile +builder. Instead of doing heuristics to compare images, it can directly track +the checksums of build graphs and content mounted to specific operations. This +makes it much faster, more precise and portable. The build cache can even be +exported to a registry, where it can be pulled on-demand by subsequent +invocations on any host. + +LLB can be generated directly using a [golang client package](https://pkg.go.dev/github.com/moby/buildkit/client/llb) +that allows defining the relationships between your build operations using Go +language primitives. This gives you full power to run anything you can imagine, +but will probably not be how most people will define their builds. Instead, we +expect most users to use a frontend component or LLB nested invocation to run +some prepared combination of build steps. + +A frontend is a component that takes a human-readable +build format and converts it to LLB so BuildKit can execute it. Frontends can +be distributed as images, and the user can target a specific version of a +frontend that is guaranteed to work for the features used by their definition. + +For example, to build a [Dockerfile](../../engine/reference/builder.md) with +BuildKit, you would [use an external Dockerfile frontend](dockerfile-frontend.md). diff --git a/build/images/buildkit-dag.svg b/build/images/buildkit-dag.svg new file mode 100644 index 000000000000..ead20f9d9eb4 --- /dev/null +++ b/build/images/buildkit-dag.svg @@ -0,0 +1,74 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/build/index.md b/build/index.md index 2d4f9178642b..1d83ed739177 100644 --- a/build/index.md +++ b/build/index.md @@ -20,29 +20,24 @@ and tools. The most common method of executing a build is by issuing a sends the request to Docker Engine which, in turn, executes your build. There are now two components in Engine that can be used to build an image. -Starting with the [18.09 release](../engine/release-notes/18.09.md#18090), Engine is -shipped with Moby [BuildKit](https://github.com/moby/buildkit){:target="_blank" rel="noopener" class="_"}, -the new component for executing your builds by default. - -BuildKit is the backend evolution from the Legacy Builder, it comes with new -and much improved functionality that can be powerful tools for improving your -builds' performance or reusability of your Dockerfiles, and it also introduces -support for complex scenarios. +Starting with the [18.09 release](../engine/release-notes/18.09.md#18090), +Engine is shipped with Moby [BuildKit](extend/index.md), the new component for +executing your builds by default. The new client [Docker Buildx](https://github.com/docker/buildx){:target="_blank" rel="noopener" class="_"}, is a CLI plugin that extends the docker command with the full support of the -features provided by BuildKit builder toolkit. `docker buildx build` provides -the same user experience as `docker build` with many new features like creating -scoped builder instances, building against multiple nodes concurrently, outputs -configuration, inline build caching, and specifying target platform. In -addition, Buildx also supports new features that are not yet available for -regular `docker build` like building manifest lists, distributed caching, and -exporting build results to OCI image tarballs. +features provided by [BuildKit](extend/index.md) builder toolkit. [`docker buildx build` command](../engine/reference/commandline/buildx_build.md) +provides the same user experience as `docker build` with many new features like +creating scoped [builder instances](building/drivers/index.md), building +against multiple nodes concurrently, outputs configuration, inline [build caching](building/cache/index.md), +and specifying target platform. In addition, Buildx also supports new features +that are not yet available for regular `docker build` like building manifest +lists, distributed caching, and exporting build results to OCI image tarballs. Docker Build is way more than a simple build command and is not only about packaging your code, it's a whole ecosystem of tools and features that support not only common workflow tasks but also provides support for more complex and -advanced scenarios: +advanced scenarios. ## Building your images @@ -116,11 +111,12 @@ to be built concurrently as part of a single request: ## Extending BuildKit -### Custom syntax on Dockerfile +### Custom Dockerfile syntax Use experimental versions of the Dockerfile frontend, or even just bring your -own to BuildKit using the power of custom frontends. See also the -[Syntax directive](../engine/reference/builder.md#syntax). +own to BuildKit using the power of custom frontends. + +[Custom Dockerfile syntax](extend/dockerfile-frontend.md){: .button .outline-btn } ### Configure BuildKit