Implement Docker caching on top of BuildKit #81

Kobzol · 2023-10-09T15:40:29Z

We cache all Docker containers built on rust-lang/rust CI on S3, to make rebuilds pretty much instant. This happens here. In August 2023, GitHub updated their Docker version, which then started defaulting to the BuildKit Docker backend. This backend doesn't support our way of exporting the Docker layer SHAs (moby/buildkit#1235), therefore after this update, our caching stopped working, which immediately made our CI workflows run for much longer.

I "fixed" this in rust-lang/rust#114763, where I just opted in to using the old Docker build backend. However, it is marked as legacy, and it will eventually stop working. When that happens, we should have some other solution for using a cache with BuildKit.

2023-10-09T06:02:41.6917009Z Attempting with retry: docker build --rm -t rust-ci -f /home/runner/work/rust/rust/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile /home/runner/work/rust/rust/src/ci/docker
2023-10-09T06:02:41.6995446Z DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
2023-10-09T06:02:41.6995904Z             BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
2023-10-09T06:02:41.6997019Z             environment-variable.

There is supposed to be a S3 cache backend for Docker with BuildKit (https://docs.docker.com/build/cache/backends/s3/), but it's marked as experimental currently, and it needs some custom Docker driver, which I'm not sure if it works on GHA. There is also GHA cache available for Docker, but it's not applicable to us, since it has a 10 GiB size limit (AFAIK).

Note: the way our caching currently works, we have to export all the intermediate Docker layers. It's not enough to just export the last layer. Because Docker performs the build layer by layer, and if it won't find an existing image for the first layer, it will just rebuild everything from scratch (in other words, it does not know the SHA of the final layer until it builds the penultimate layer).

The text was updated successfully, but these errors were encountered:

Kobzol · 2023-10-18T20:29:41Z

Trying to fix this in rust-lang/rust#114762. So far I'm having problems with the BuildKit S3 cache backend not having the correct permissions to access an S3 bucket.

Mark-Simulacrum · 2023-10-25T01:13:26Z

https://aws.amazon.com/blogs/containers/announcing-remote-cache-support-in-amazon-ecr-for-buildkit-clients/

This might also be interesting to explore in the future.

Kobzol · 2023-12-25T09:09:58Z

I implemented caching using the ghcr.io registry here. It stores the cached Docker images under ghcr.io/rust-lang-ci/rust-ci. It seems to work for try/auto builds, PR builds don't have the necessary permissions on their GITHUB_TOKEN, which is probably a good thing, and it has also worked like this previously.

…try> Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

…try> Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

…try> Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Rustup part of this change is [here](rust-lang/rustup#3648). Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

…ark-Simulacrum Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Rustup part of this change is [here](rust-lang/rustup#3648). Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

…ark-simulacrum Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Rustup part of this change is [here](rust-lang/rustup#3648). Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

…acrum Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Rustup part of this change is [here](rust-lang/rustup#3648). Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

Kobzol · 2024-02-13T16:03:46Z

Implemented in rust-lang/rust#119290.

…acrum Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Rustup part of this change is [here](rust-lang/rustup#3648). Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

jdno added this to infra-team Oct 9, 2023

github-project-automation bot moved this to Backlog in infra-team Oct 9, 2023

Kobzol mentioned this issue Dec 13, 2023

Create a GitHub token for publishing (Docker) packages #95

Closed

Kobzol mentioned this issue Dec 25, 2023

Cache CI Docker images in ghcr registry rust-lang/rust#119290

Merged

Kobzol closed this as completed Feb 13, 2024

github-project-automation bot moved this from Backlog to Done in infra-team Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Docker caching on top of BuildKit #81

Implement Docker caching on top of BuildKit #81

Kobzol commented Oct 9, 2023 •

edited

Loading

Kobzol commented Oct 18, 2023

Mark-Simulacrum commented Oct 25, 2023

Kobzol commented Dec 25, 2023

Kobzol commented Feb 13, 2024

Implement Docker caching on top of BuildKit #81

Implement Docker caching on top of BuildKit #81

Comments

Kobzol commented Oct 9, 2023 • edited Loading

Kobzol commented Oct 18, 2023

Mark-Simulacrum commented Oct 25, 2023

Kobzol commented Dec 25, 2023

Kobzol commented Feb 13, 2024

Kobzol commented Oct 9, 2023 •

edited

Loading