GitHub Actions objects mode: the exported bundle keeps predictions from the job's last mbx command only, so a multi-command job restores nothing next time

[jeremy]

Summary

With jdx/mr-boxington-action in github-cache-mode: objects, a job that runs several mbx commands exports a bundle whose task manifest holds the predictions of the last command only. The next job restores the bundle (every object and action result is there), but its first commands find no prediction for their compilations, report N not looked up, and compile everything cold. We saw this on GitHub-hosted runners and then reproduced it on a Linux workstation in the same shape; the cause is visible in export_receipts, and the fix looks small.

What we saw in CI

Public repo (basecamp/hey-sdk; the trial is #182, the write-up and revert #187), Rust Tests job: rustup toolchain install, the action (v1.3.0, version: 1.10.0, github-cache-mode: objects), then make rs-check CARGO=mbx, which runs mbx fmt, mbx clippy --workspace --all-targets --all-features, mbx test --workspace --all-features, mbx doc, mbx clippy … --no-default-features, mbx test … --no-default-features, mbx build --examples, cargo deny, mbx publish --dry-run. Entries are saved from pushes to main only; pull requests restore.

A pull request that left rust/ untouched restored main's entry with an exact key hit (imported 1082 actions and 4860 objects … (10.5 GiB), about 80 s to download and import), and then:

mbx[cache]: object cache: 2 hits, 0 misses, 654 not looked up, 10 bypassed; …   (clippy, 66 s)
mbx[cache]: object cache: 4 hits, 0 misses, 605 not looked up, 11 bypassed; …   (test, 138 s)
mbx[cache]: … 19 not looked up … 58 not looked up … 60 not looked up … 6 … 15 …  (the rest)
mbx[cache]: 654 compilations had no usable prior inputs or matching prediction for a lookup

Rust Tests went from 193–204 s (rust-cache hit) to 394–430 s on pull requests and 455–468 s on pushes to main (cold build plus a 122–183 s post step saving a 2.5 GB entry). This run was made with the store sweep disabled (MBX_GC_AUTO=0), so eviction is not the explanation; see the note at the end for what happens with the sweep on.

Reproduction on a Linux workstation (mbx main at 7919cf4, rustc 1.97.1, btrfs)

Same public repo at 23887598. Populate a fresh store from a fresh clone with MBX_CACHE_EXPORT_GROUP set, mbx cache export --group …, import the tar into an empty store, delete and re-clone the checkout at the same path, run the commands again.

Populate commands (in order)	bundle	after import: clippy --all-targets --all-features	after import: test --all-features --no-run
clippy only	484 actions, 1 task manifest with 484 predictions	273 hits, 3 misses, 184 not looked up, 5.1 s	2 hits, 605 not looked up (never populated; expected)
clippy, then test --no-run	892 actions, 1 task manifest with 424 predictions	55 hits, 1 miss, 404 not looked up	273 hits, 2 misses, 184 not looked up, 6.1 s

The second row is the CI shape in miniature: the bundle carries every action from both commands (892) but only the second command's predictions (424), so the first command after import cannot look anything up except the actions the two commands share (build scripts, proc-macro crates). The 184 "not looked up" in the hit rows are aws-lc-sys C compilations (cc-unportable-output in MBX_BYPASS_LOG), unrelated to this.

In CI the last command that touches the workspace lockfile is a small one (build --examples, or the no-default-features test build), so the manifest that ships carries 6–60 predictions and the next job's clippy pass, with 654 compilations, finds none. A local store on the same workstation hits (478 hits on a second clone) because it learned every command's predictions itself.

Where it comes from

crates/mbx-cache-store/src/lib.rs, export_receipts: receipts are sorted by completion time and inserted into a map keyed by receipt.identity, so a later receipt replaces an earlier one:

// One task manifest can carry only one action per invocation. The
// newest run of the same task is the useful prediction set to import;
// every older action remains in `actions` and therefore in the bundle.
tasks.insert(receipt.identity.clone(), TaskActionManifest { …, predictions: receipt.predictions });

The identity is the digest of Cargo.lock (build_identity → workspace_marker), so every mbx command in a job shares one identity, and a receipt holds "the exact cache predictions completed by one top-level build command". Two commands with different flags predict different invocations, so the newest receipt is not a superset of the older ones; the comment's assumption holds for repeated runs of one command, not for a job that runs several.

On the import side, merge_imported_manifest already unions by invocation when a manifest for the identity exists (imported predictions first, existing ones fill in). Doing the same at export — fold every receipt for an identity into one manifest, newest prediction per invocation — would give the next job a lookup for each command the previous job ran. An alternative that needs no format change would be for the action to run one mbx cache export per command, but the bundle is one GitHub cache entry, so the union at export seems like the right place.

Note on the store sweep

With the sweep left at its default, gc.max_size is 5% of the runner's disk, well under the 10.5 GiB the import had just written, and the first mbx command of the job evicted about half of the imported objects before compiling. MBX_GC_AUTO=0 for the job fixed that. It may be worth the action either disabling the automatic sweep for the job or raising the budget to at least the bundle it just imported.

Happy to test a branch; the repository, commit and commands are all public, and the workstation reproduction takes about a minute.

[jdx]

Confirmed—your diagnosis matches the export path. I've opened #424 to union each task's predictions across all command receipts, choosing the newest completed receipt only when the same invocation appears more than once. The archive still retains every referenced action, including superseded results.

The regression exports and imports overlapping command receipts into an empty store. It fails on the old code and passes with the fix; all 50 cache-store tests passed on Linux, and the full local CI gate passed on macOS.

The GC budget issue is separate and isn't changed by this PR. MBX_GC_AUTO=0 remains the workaround for your disposable runner experiment; this fix prevents missing predictions but doesn't prevent eviction when the imported cache exceeds the configured budget. The Action will need an mbx version containing the fix before its normal installation picks it up.

[jdx]

I've opened a separate Action fix for the GC-budget problem: jdx/mr-boxington-action#37.

For GitHub-hosted runners using github-cache-mode: objects, the Action will default MBX_GC_AUTO=0 for the job unless you've explicitly set it. Self-hosted/unknown runners retain their existing policy; disposable self-hosted jobs can set MBX_GC_AUTO=0 at job level. Hosted jobs can opt back into cleanup with MBX_GC_AUTO=1. Disabling cleanup lets the cache grow during the job.

The regression makes the timing concrete: import a bundle, build with a 1-byte budget and collection due, then export the completed command's closure. With GC enabled the first build hits, but the post-build sweep evicts the objects and re-export fails with action result is missing. With GC disabled the hit and re-export both succeed. Type checking, all 37 Action tests, and the generated bundle check pass; the regression is also wired into hosted Linux/macOS/Windows CI.

This complements #424's prediction-union fix; it doesn't require a new mbx feature. Until the Action change is released, the job-level MBX_GC_AUTO=0 workaround remains applicable.

[jeremy]

Results from CI with the fix, on the same hey-sdk Rust Tests job as above (github-cache-mode: objects, MBX_GC_AUTO=0 at job level, new cache-generation). Because the action's version input takes releases only, the job installed mbx from main at 68e48f1 (#424 included) with cargo install --git … --rev, cached with actions/cache (213 s once, 5 MB entry); this was before v1.10.1 shipped.

Push to main (cold, saves the entry): Rust Tests 525 s = 213 s one-time mbx build + 249 s rs-check (cold) + 5 s drift + 39 s post-step export — 1082 actions / 4860 objects, 5.0 GiB raw → 1221 MB entry (was 2540 MB and 122–183 s to save). Without the mbx build, ~312 s against the 455–468 s pushes in the report.

Pull request with rust/ untouched (exact key hit): Rust Tests 148 s = 32 s restore (5 s download, 11 s unpack, 16 s import) + 96 s rs-check:

clippy  --all-targets --all-features   485 hits, 1 miss, 184 not looked up   36.8 s
test    --all-features                 426 hits, 1 miss, 184 not looked up   37.7 s
doc / clippy nd / test nd / examples / publish --dry-run   21 / 58 / 60 / 6 / 15 hits

against 193–204 s with rust-cache and 394–430 s with the last-command-only manifest, so the prediction union is exactly the difference. The 184 not looked up are the aws-lc-sys C set (178 cc-unportable-output plus the six changing-directory/oversized units); under 1.10.1 (#423) the 178 become path-bound entries, and since a hosted runner's checkout path is stable across jobs they should start hitting there too — we'll see it on the next generation.

We're pinning version: 1.10.1 now (the digest resolves from the release asset, so the released action handles it), and keeping the job-level MBX_GC_AUTO=0 until an action release carries #37. Thanks for #424 and #37 — both landed the same day as the report.