ci: cache apt packages between runs - #141
Merged
Merged
Conversation
apt-get was 15-37s in every job. On the short legs it is most of the wall time
- build (xfs, gcc) spent 28s of 33s installing packages and 5s testing - and
it is ~14% of the valgrind leg, which sets CI's critical path.
Each job now restores ~/.apt-archives with actions/cache before installing.
The key pins the runner image and the hash of the package list, so a hit means
the same packages resolved against the same image; the installer then runs
dpkg straight from the cached .debs, with no apt-get update and no network.
apt caches dependencies it downloaded too, so the set is self-contained.
The package lists move to .github/apt/*.txt, one per leg. They have to be
files rather than inline shell for the cache key to hash them, and the three
sets are unchanged - verified token by token against the previous workflow.
Two things the fast path has to get right:
- dpkg succeeding does not mean the build has what it needs. A partial cache
can install cleanly and still leave a package missing, which would surface
later as a confusing compile error, so the requested packages are checked
explicitly with dpkg-query and any shortfall falls back to apt-get.
- apt drops privileges to _apt to download, so the archive dir is chowned to
it before the install and back to the runner user afterwards, since
actions/cache saves as that user.
Every path falls back to plain apt-get on any doubt - cache miss, stale cache,
or a dpkg that will not take the set - so the worst case is the speed we had
before, not a broken job.
Tested against Ubuntu 24.04, the same family as the runners: cold cache
populates and installs; warm cache installs from disk in 0.2s; a deliberately
partial cache is detected and recovered via apt-get.
Expect no change on the first run of this - it populates the cache - and
roughly 25s off each job after that. CI wall-clock is set by the valgrind leg,
so the end-to-end gain is ~10%; the rest is runner-minutes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxN8SYwbj24nAyAAsPRdm4
Follow-up to the apt cache, from a review pass. One real defect and a lot of
duplication.
The key pinned nothing. It read ${{ env.ImageOS }}/${{ env.ImageVersion }},
which are set in the runner's environment but are not visible to workflow
expressions, so the first run saved `apt---build-<hash>` - image-blind, and a
runner upgrade would have reused .debs built against the old one. They are now
read in a shell, where they do exist, with /etc/os-release as a fallback so the
key can never silently lose the image again.
The three package lists shared 9 of their ~12 entries, so 27 of 35 lines were
the same set written three times, and adding a dependency meant getting it
right in three places or breaking one leg as a confusing compile error. Now
base.txt plus small per-leg extras, with the installer taking several lists.
Each leg installs exactly what it did before, verified set by set.
The cache+install step pair was pasted into three jobs, rationale comment and
all. It moves to .github/actions/apt-deps, so each job is one `uses:` line and
the key expression exists once - which is also what makes the image-id step
above affordable, rather than a fourth copy.
Smaller things from the same pass: the second all_present() after apt-get was
unreachable (apt-get already exits non-zero under set -e); `apt-get install -f`
ran on the cold path where nothing can be broken, and moves to the only branch
whose comment describes it; the .deb set was enumerated three ways where a
nullglob array does it once; all_present() forked twice per package instead of
querying once; and the header lost the measurements that belong in a commit
message. Shebang and message prefixes now match scripts/verify.sh.
Not taken: a single union list (each leg would install valgrind/llvm/xfsprogs
it does not need, to save cache storage that is nowhere near the 10GB limit);
restore-keys for partial restores (needs the fast path gated on an exact hit to
be worth it); and converting release.yml's fourth copy of the list, which is a
different workflow. Sharding the valgrind leg is the real remaining lever and
wants its own change.
Retested on Ubuntu 24.04 across all three paths: cold populates, warm installs
from disk, partial cache is detected and recovered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxN8SYwbj24nAyAAsPRdm4
martinus
force-pushed
the
claude/ci-apt-cache
branch
from
July 25, 2026 19:18
90081b4 to
1a8566f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #139. With the suite parallel,
apt-getis now a visible share of every job: 15-37s, which on the short legs is most of the wall time —build (xfs, gcc)spent 28s installing packages and 5s testing — and ~14% of the valgrind leg that sets CI's critical path.Each job restores
~/.apt-archiveswithactions/cachebefore installing. The key pins the runner image and the hash of the package list, so a hit means the same packages resolved against the same image. The installer then runsdpkgstraight from the cached.debs— noapt-get update, no network. apt caches the dependencies it downloaded too, so the set is self-contained.Package lists move to
.github/apt/*.txt, one per leg. They have to be files rather than inline shell for the cache key to hash them. The three sets are unchanged, verified token by token against the previous workflow.Two things the fast path has to get right
dpkgsucceeding does not mean the build has what it needs. A partial cache can install cleanly and still leave a required package missing, surfacing later as a confusing compile error. The requested packages are checked explicitly withdpkg-query, and any shortfall falls back toapt-get._aptto download, so the archive dir is chowned to it before the install and back to the runner user afterwards, sinceactions/cachesaves as that user. Without this apt warns and refetches as root.Every path falls back to plain
apt-geton any doubt — cache miss, stale cache, or adpkgthat will not take the set — so the worst case is the speed we have today, never a broken job.Testing
Run against Ubuntu 24.04, the same family as the runners, exercising all three paths:
Expected effect
No change on the first run — it populates the cache. After that, roughly 25s off each job. Since CI wall-clock is set by the valgrind leg, end-to-end gain is ~10%; the rest shows up as runner-minutes rather than latency.
Worth being clear that this is the smaller of the two remaining levers. Valgrind's integration step is still 182s of its 226s, and that — not apt — is what CI wall-clock is now made of.
Generated by Claude Code