chore(ci): evict stale lake-packages cache in host maintenance#2037
Merged
Conversation
The weekly host-maintenance run prunes lake-build, compiler-ccache, journald, and docker, but never touched /srv/verity-ci-cache/lake-packages. That dir grows one ~5.7G entry per PR/branch/content-hash and was never evicted, so the x64 runner root disks filled to 100% and the runners silently dropped offline (the runner service stays active but can't write). Add an mtime-based prune for lake-packages (default 14 days, override via LAKE_PACKAGES_MAX_AGE_DAYS), mirroring the existing lake-build eviction. Worst case of an over-prune is a cache miss on the next build, never incorrectness.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/ci_host_maintenance.sh run(weeklyverity-ci-host-maintenance.timer, Sun 04:30) pruneslake-build,compiler-ccache, journald, and docker — but had no rule forlake-packages./srv/verity-ci-cache/lake-packagesgrows one ~5.7G entry per PR/branch/content-hash and was never evicted. On 2026-06-18/19 both x64 CI runner root disks hit 100% (hz2 701G/125 entries, hz1 1.1T/194 entries, some back to April) and the runners silently dropped offline — the runner systemd service staysactive runningbut can't write to a full disk, so it just disconnects from GitHub.lake-packages(default 14 days, override viaLAKE_PACKAGES_MAX_AGE_DAYS), mirroring the existinglake-buildeviction and reusing the sameprune_treehelper.Risk
lake-build/compiler-ccacheevictions.Test plan
bash -n scripts/ci_host_maintenance.sh(syntax)LAKE_PACKAGES_MAX_AGE_DAYS=14 scripts/ci_host_maintenance.sh runprintslake-packages cache: removed N entries older than 14 daysand frees diskinstall-systemdso the weekly timer picks up the new prune (no unit change required; therunsubcommand reads the updated script)Note
Low Risk
Operational disk cleanup only; over-pruning would cause slower rebuilds (cache miss), not incorrect builds—same profile as existing
lake-buildeviction.Overview
Weekly
scripts/ci_host_maintenance.sh runnow prunes$CACHE_ROOT/lake-packagesthe same way it already pruneslake-buildandcompiler-ccache, using the existingprune_treehelper (top-level entries older than mtime threshold).A new env knob
LAKE_PACKAGES_MAX_AGE_DAYSdefaults to 14 and is documented inusage(), alongside the existing cache age variables.Reviewed by Cursor Bugbot for commit bc5f015. Bugbot is set up for automated code reviews on this repo. Configure here.