CI: switch to old docker builder to temporary solve caching issue #114621
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
(rustbot has picked a reviewer for you, use r? to override) |
rustbot
added
A-testsuite
|
Oh, yes, i've tried to solve caching issue, but for PR jobs it didn't worked before too. |
|
@bors try |
|
⌛ Trying commit 5592aa3 with merge c4e6680c1530436eac185b091edd9f1a292520c2... |
|
This isn't working currently. Docker updated from 20.10.25+azure-2 to 23.0.6+azure-2, changing it's behavior and switching builder to buildx. Problem us that previously builded image has all image ids in history, so doing docker history -q rust-ci | \
grep -v missing | \
xargs docker save | \
gzip | \
$uploadsaved all layers, but now it's empty all, except last one: Either restoring missing ids or playing with inline cache can fix things. |
|
☀️ Try build successful - checks-actions |
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Aug 12, 2023
…-Simulacrum CI: fix Docker layer caching As reported by `@klensy` on [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/docker.20images.20always.20rebuilded), Github Actions have recently updated their Docker version from 20.x to 23.x, which enabled the BuildKit build backend by default. This broke our way of performing Docker layer caching on CI, which immediately made all non-PR CI builds (including try builds) ~1 hour longer (Docker caching didn't work on PR builds before, so it wasn't affected). The moment this started happening can be seen [here](https://github.com/rust-lang-ci/rust/actions?page=2&query=branch%3Aauto+is%3Asuccess). The problem is with the following command: ``` docker history -q rust-ci | \ grep -v missing | \ xargs docker save | \ gzip | \ $upload ``` which returns the intermediate layers as `<missing>`, if BuildKit is enabled. This was investigated by `@klensy` in rust-lang#114621. Thanks for that! I will continue experimenting with how we can enable the cache with BuildKit in rust-lang#114762, but for the time being, I think that we should just hotfix this. This PR reverts the build backend back to the old one, which fixes the caching. However, we also have to bust the cache of all Dockerfiles, otherwise caching would only start kicking in for them the next time they are updated (or the next time GH updates their docker version). Because when the Docker version was updated the last time, the Dockerfiles were cached on S3 with basically an empty cache, and unless we bust it, even after reverting to the old build engine, the CI script would just download the empty cache and rebuild the Dockerfile from scratch, thus nullifying our fix. r? `@Mark-Simulacrum`
GuillaumeGomez
added a commit
to GuillaumeGomez/rust
that referenced
this pull request
Aug 13, 2023
…acrum CI: use smaller machines in PR runs mingw-check job-linux-16c -> job-linux-4c ~job-linux-4c 20 min in auto job ~job-linux-16c 13 min in pr job with current pr regressed to almost 21 min, it's ok. mingw-check-tidy job-linux-16c -> job-linux-4c small enough, so reduce to minimal ~ job-linux-16c 3 min with current pr regressed to almost 5 min, it's ok. x86_64-gnu-tools job-linux-16c this is top job by time in PR, so don't touch it ~ job-linux-8c 1.30 hour in auto job ~ job-linux-16c 1 hour in pr job (affected by rust-lang#114613, actual time ~ 30 min) x86_64-gnu-llvm-15 job-linux-16c don't change too ~ job-linux-8c 1.30 hour in auto job ~ job-linux-16c 30 min in pr job Noticed while working on rust-lang#114621, so current time affected by always rebuilded docker images (but pr images always rebuilded before too, so nvm)
|
Okay, will close @Kobzol applied fix. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switch to old docker builder to temporary solve caching issues.
Also silence unzip, it too noisy