Make Core CI Faster With Caching and Less Artifact Traffic

[chet]

TL;DR

Core CI repeats a lot of expensive Rust and artifact work, and the shared compiler cache in #4513 has not produced a reusable hit yet. Keep the parts #4513 proves are helpful, then fix or remove the broken cache path and only claim cache savings once one run writes entries that a later run reuses.

Details

Core CI has been spending close to an hour building the same classes of Rust and boot outputs across separate jobs. PR #4513 already starts independent lint and release work together, narrows several large artifact downloads, and adds caches intended to reuse work between runs.

The first measured result is useful, but one advertised part is not working yet. The latest successful Core run reached the required check in 48m31s, while the shared Rust compiler cache reported zero hits and failed every remote write on both architectures. This issue brings that existing PR into the epic, keeps the working improvements, and requires the cache and permission claims to match what the logs actually prove.

Expected impact

• Expected green-run effect: The latest successful Core run reached core-ci-pass in 48m31s, which is 14m43s faster than the earlier 63m14s median. Treat that as one provisional result, not proof of the advertised roughly 25-minute target; the comparable REST run remained near its existing 14-minute baseline.
• What it really buys us: It starts independent lint and release-build work together, narrows several large artifact downloads, and adds cache plumbing we can measure. The shared Rust compiler cache did not contribute to the measured improvement because that run recorded zero hits and roughly 1,500 write errors on each architecture.
• Metrics affected: Core author-feedback time, Core runner-minutes, artifact upload/download bytes and time, BuildKit and mkosi cache behavior, and Rust compiler-cache hits, misses, and write errors.
• Confidence: Medium for the observed overall improvement because it is one current run compared with a historical median; low for sustained cache savings until a successful main write is followed by real hits on a later PR.
• Applies to: Core runs that execute the full build and artifact lane. It does not materially improve a REST-only run or a Core run whose artifact work is later unselected.
• How we'll measure it: After merge, compare at least six successful, final-head full Core runs with the six-run baseline. Report required-check wall time, total and build-* runner-minutes, transferred bytes, and each cache's hit/write statistics separately. Do not count Rust compiler-cache savings until a main run writes successfully and a later PR restores actual hits.

What this involves

• Keep the measured lint/release-build overlap and architecture-specific artifact selection from PR ci: Optimize Core CI using increased actions cache quota, parallelization #4513.
• Fix the shared compiler-cache 404 write failures, or remove that cache path and its projected savings from the PR. A cold cache may have zero hits, but it must successfully write entries before a later run can reuse them.
• Keep BuildKit cache export restricted to trusted main runs, and record which PR contexts may read or write each other cache.
• Give the affected PR jobs explicit least-privilege token permissions. Remove unnecessary secrets: inherit, and inventory the new Actions runtime/results credential passed into PR-controlled BuildKit work so CICD-02 and CICD-13 start from the correct trust boundary.
• Record exact uploaded and downloaded bytes by producer and consumer. Do not use the stored-artifact total alone as proof of the PR's larger traffic claim, especially when compression settings changed.
• Record at least one successful main cache population and a later PR with actual hits before crediting warm-cache time.
• Preserve every existing required check and artifact output that downstream jobs still consume. CICD-04 and CICD-05 own the remaining exact-name, manifest, and fail-closed work.
• Before merge, link this issue from PR ci: Optimize Core CI using increased actions cache quota, parallelization #4513, replace Related issues: N/A, and normalize the branch to one signed commit under the repository contribution workflow.

Part of #4572.