Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: fix Docker layer caching #114763

Merged
merged 1 commit into from
Aug 12, 2023
Merged

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Aug 12, 2023

As reported by @klensy on Zulip, Github Actions have recently updated their Docker version from 20.x to 23.x, which enabled the BuildKit build backend by default.

This broke our way of performing Docker layer caching on CI, which immediately made all non-PR CI builds (including try builds) ~1 hour longer (Docker caching didn't work on PR builds before, so it wasn't affected). The moment this started happening can be seen here.

The problem is with the following command:

docker history -q rust-ci | \
          grep -v missing | \
          xargs docker save | \
          gzip | \
          $upload

which returns the intermediate layers as <missing>, if BuildKit is enabled. This was investigated by @klensy in #114621. Thanks for that!

I will continue experimenting with how we can enable the cache with BuildKit in #114762, but for the time being, I think that we should just hotfix this.

This PR reverts the build backend back to the old one, which fixes the caching. However, we also have to bust the cache of all Dockerfiles, otherwise caching would only start kicking in for them the next time they are updated (or the next time GH updates their docker version). Because when the Docker version was updated the last time, the Dockerfiles were cached on S3 with basically an empty cache, and unless we bust it, even after reverting to the old build engine, the CI script would just download the empty cache and rebuild the Dockerfile from scratch, thus nullifying our fix.

r? @Mark-Simulacrum

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Aug 12, 2023
@Mark-Simulacrum
Copy link
Member

@bors r+ rollup=never p=1

@bors
Copy link
Contributor

bors commented Aug 12, 2023

📌 Commit 6ca13d0 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 12, 2023
@bors
Copy link
Contributor

bors commented Aug 12, 2023

⌛ Testing commit 6ca13d0 with merge 28eb857...

@bors
Copy link
Contributor

bors commented Aug 12, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 28eb857 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Aug 12, 2023
@bors bors merged commit 28eb857 into rust-lang:master Aug 12, 2023
12 checks passed
@rustbot rustbot added this to the 1.73.0 milestone Aug 12, 2023
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (28eb857): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-0.4% [-0.4%, -0.4%] 2
All ❌✅ (primary) - - 0

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 634.075s -> 632.358s (-0.27%)

@Kobzol Kobzol deleted the fix-ci-docker-caching branch August 13, 2023 07:18
@Kobzol
Copy link
Contributor Author

Kobzol commented Aug 14, 2023

CI is fast again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants