feat(scratch): one cache lineage per branch, seeded on a switch - #431
Closed
raphaelvigee wants to merge 2 commits into
Closed
feat(scratch): one cache lineage per branch, seeded on a switch#431raphaelvigee wants to merge 2 commits into
raphaelvigee wants to merge 2 commits into
Conversation
raphaelvigee
force-pushed
the
raphaelvigee/scratch-mount
branch
from
August 29, 2026 15:34
4573c40 to
1625864
Compare
raphaelvigee
force-pushed
the
raphaelvigee/scratch-lineage
branch
from
August 29, 2026 15:34
e5ed599 to
3e245e0
Compare
raphaelvigee
marked this pull request as ready for review
August 29, 2026 15:34
…share
Every sandbox starts empty. That is the hermetic default and it is correct, but
it means heph re-pays, on every miss, for work that is byte-identical across
targets and across runs — compiler caches, dependency caches, tool indexes.
# //build/BUILD — declared once
target(name = "gocache", driver = "scratch",
path = ".cache/go-build", env = "GOCACHE", access = "shared")
# any number of targets, wiring nothing
target(name = "server", driver = "bash", scratch = ["//build:gocache"],
run = ["go build ./..."]) # GOCACHE is set for the run
heph already hit this once and solved it by hand inside one driver:
`plugin-go`'s shared golist GOCACHE. Measured on a 500-package corpus, `go list`
was 778s of CPU across 1945 invocations (687s of it *system* time) against 15s
for every `go tool compile` combined; sharing the cache took the run from 205s to
84s wall. That module has no lock, no eviction, no remote and no visibility, and
is available to exactly one driver.
## The contract
> A target must produce identical outputs whether its scratch directories are
> warm, cold, or absent. **Losing one is always a slowdown and never a wrong
> answer.**
Everything below follows from it. This is the same promise Go's build cache,
ccache and sccache already make; it does *not* hold for a directory used as
durable state, and the docs say so in those words.
## A target, not an attribute
Declaring the settings inline at each use site would make every consumer restate
`access`, `version` and `remote`, then need validation that they all agree — a
*discovered* error where a declaration makes it inexpressible. There is now
exactly one copy of each. Cache identity is the addr, so packages namespace it
for free: `//go:cache` and `//rust:cache` are different caches with nobody
agreeing on a prefix convention.
No new Starlark global: heph's rule surface is `target(driver = "…")`, so this is
a builtin driver shaped like `plugingroup`, and `#[derive(Spec)]` supplies both
the parser and the LSP schema.
## Nothing about a scratch reaches `hashin`
A reference is an `Input` with `hashed: false, runtime: false` — the one
combination nothing else uses. It materializes no artifacts, and it must not
touch the consumer's cache key.
The tempting alternative is to fold the declaration in, so bumping `version`
rebuilds users of the cache. That is an over-hash: if outputs really are
identical warm-or-cold, a fresh slot changes nothing and the rebuild is pure
waste; if they are not, the target is already broken and rebuilding is not the
fix. So a `version` bump gives every consumer a fresh empty slot and invalidates
nothing — exactly what you want when the reason for bumping is "the old cache
went bad". Asserted against the **def hash**, not `hashout`: a target whose key
moved still produces identical bytes, so a hashout comparison would pass while
the cache missed on every run.
## Mounting is one symlink
Per target, pointing out of the sandbox at the canonical slot directory. Teardown
removes the link, not the tree (`remove_dir_all` does not follow symlinks) — the
same property read-only input staging already relies on for the 11k-file Go SDK.
The measurements rule out the alternative: seeding a warm cache *into each
sandbox* cut `go list` CPU 778s -> 309s and moved wall time by **exactly zero**,
because the cost was never CPU but the ~500 filesystem entries created and
destroyed per sandbox.
The link target is the *canonical* path, not something sandbox-local. Tools bake
absolute paths into their cache entries, so if every consumer saw its own path
the cache would restore and be inert — present, and useless.
The bridge creates it rather than the engine, because the bridge owns sandbox
creation: the FUSE path may redirect the package dir into a mount, so there is no
earlier moment at which the directory reliably exists.
## The one wrong-build guard
A scratch that would land where an input already did is a hard error. It is the
only way a scratch can cause a wrong build rather than a slow one — the target
would read cache contents where it believes it reads a declared dependency,
bytes no `hashin` describes.
## Two author assertions, both defaulting conservatively
- `access = "shared"` says the tool is safe under concurrent access. Go's build
cache is the motivating case — it is what `go build -p N` does — so forcing it
exclusive would serialize a whole Go build. Defaults to `exclusive`, and the
keyed cross-process lock ships in this change, because an `exclusive` that does
not serialize is a silent lie. Guards are taken in sorted slot order (so two
targets naming the same pair in opposite orders cannot deadlock) and acquired
after dep resolution but before the worker permit (after deps, or a dep needing
the same slot could never get it; before the permit, so a queued target holds
no worker and the wait is provably bounded).
- `platform = "any"` says the contents are portable, which lets one slot serve
every machine. It asserts two things: no host dependence *and* no embedded
absolute paths. Defaults to `os_arch`, because restoring a host-specific cache
onto the wrong host is the one mistake here that is not merely slow.
## Compatibility
`ABI_SEMVER` 0.7.0 -> 0.8.0. `RunRequest`/`ManagedRunRequest` gain
`repeated ScratchMount scratch`: additive and cold-path — a prost wire field, not
a vtable change — so an old plugin decodes a new host's request and ignores the
mounts. Its targets then run without a scratch, which costs a cold cache and
never a wrong build: the lock is keyed on a declaration an old plugin cannot see,
so there is no shared directory for it to race either.
Declaration and reference need no ABI surface at all; only mounting does. The
reference rides on `Input.annotations`, already the producer->host channel
(`read_only`/`stage_per_file` are the precedent).
Design doc: docs/SCRATCH.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Three things a scratch cache needs before it is usable in anger: it must survive
branch switching, something must bound its growth, and it has to reach CI.
## One lineage per branch
Without it, `git checkout feat-x` silently hands the arriving branch whatever
cache state the departing one left, then hands it back mutated — every switch
degrading both. It is the same event as a PR job running against a shared cache,
and it wants the same answer.
scratch:
scope: ${git:branch} # the lineage this run writes to
restoreScopes: [master] # what it may read from when its own is empty
seedOnFork: true
Reads try the current scope, then each fallback in order. **Writes only ever go
to `scope`** — never to a fallback, not even the one just seeded from. That
asymmetry is the whole of the isolation: a PR build cannot advance the cache its
base builds from, and a broken experiment on a branch cannot corrupt the one you
go back to.
`${git:branch}` is expanded by reading `.git/HEAD` directly rather than shelling
out — this runs on every engine construction, and a subprocess for one line of a
file that is always there is not worth it, nor is requiring `git` on PATH.
Outside a checkout, or on a detached HEAD, it resolves to the empty scope: one
shared lineage, today's behaviour. A cache policy must never be why a build
cannot start. Scopes are sanitized to one path component, because branch names
contain `/` and a raw one would make two branches collide the moment one is a
path prefix of another.
Seeding is the only copy in the design: once per (slot, scope), amortized over
every later build on that branch, and measured against a *cold rebuild* rather
than against nothing. Defaults on, because scoping without it makes every switch
cold — worse than not scoping at all. A failed seed is a cold cache and never a
failed build, and leaves no partial tree for the next run to mistake for a warm
one.
Defaults are unchanged: `scope` is empty, so a workspace that configures nothing
keeps the single lineage it has today.
## Nothing else bounds a scratch
It is keyed by a declaration rather than an input hash, so there is no `hashin`
to age out and no `cache.history` to trim against — it grows until something
removes it.
heph tool scratch ls / path / rm / push / pull
heph tool clean --scratch
heph tool gc --scratch-max-size 50GiB --scratch-max-age-days 30
The sweep is in `heph tool gc` rather than behind a flag nobody sets: a GC that
reclaimed target revisions and left the one part of the store that grows without
limit would be misleading about what it does. **Whole caches, never partial
trims** — heph cannot know which of a foreign tool's entries are hot, and
guessing would quietly degrade a cache while claiming to manage it.
Each slot records what it came from in a small `slot.meta`, so the store
describes itself and stays listable and removable after the target that made it
is gone. A slot whose meta is unreadable is still listed and still removable by
`--all`, but never matched by name — it cannot be named. Sizes are measured per
scope, so they mean the cache's own footprint and not that plus heph's
bookkeeping.
## `--scratch=off` forces a rebuild, and must
`off` runs every target with its caches absent — the audit mode for the contract.
The whole path is skipped, not merely mounted-and-empty.
It implies `--force`. Scratch deliberately never reaches `hashin`, so without a
rebuild an off-run is a plain cache hit that replays the result built *with* a
warm cache — the audit would pass by reading exactly the answer it is supposed to
re-derive. The vacuous case is asserted directly so the implication cannot be
simplified away later.
## The remote: pull is automatic, push is a command
A pull is read-only, costs one list plus one fetch, and every way it can fail
degrades to a cold build. So a build does it on its own, and a cold CI runner
warms itself with no workflow change.
A push is none of those, and whether a job's cache state deserves to become the
branch's published head is a CI-policy question — answered far better by an `if:`
in a workflow than by a heuristic inside heph. So:
heph tool scratch push --all --producer "$GITHUB_RUN_ID"
**Why "latest" is not a pointer.** One cache serves many branches at once — a
lineage for master, one per open PR, all advancing concurrently — so there is no
single latest for a pointer to name. A pointer *per* branch creates an unbounded
set of mutable objects with no way to relate the heads a cross-branch restore
must compare. And the store could not maintain one anyway: `RemoteCacheBackend`
has open_read/open_write/exists/list_names — no compare-and-swap and no delete,
so two jobs finishing together race and the loser can be the one that finishes
last, overwriting the pointer with older content.
So entries are immutable and ordering is in the key. The generation is
zero-padded hex and leads the key, so lexicographic order *is* generation order
and a listing sorts without fetching anything. A publish is `parent + 1` within
its lineage — deliberately not a clock: a slow runner that picked up generation 5
an hour ago and finishes now publishes 6, which correctly loses to a chain that
has since reached 12.
A same-generation fork is expected, not an error; the tie-break only has to be
deterministic so readers converge, and ordering by `(generation, bytes, key)`
makes it useful too, preferring the fuller cache. Republishing identical contents
is skipped — the archive is deterministic, so unchanged contents hash
identically, and without this a chain would grow on every no-op CI run.
Symlinks are archived as symlinks rather than followed, so a slot that acquired a
link out of the tree does not publish whatever it points at to every machine that
picks the snapshot up. A snapshot records the path it was produced at, because a
path-sensitive cache restores fine elsewhere and is then *inert* — present and
useless, which looks exactly like a hit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
raphaelvigee
force-pushed
the
raphaelvigee/scratch-lineage
branch
from
August 29, 2026 15:51
3e245e0 to
c61ce8f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth of four (stack #408), on top of #412. The first three made a scratch cache
work; this one makes it survive branch switching.
The problem
With one lineage per slot,
git checkout feat-xsilently hands the arrivingbranch whatever cache state the departing one left, then hands it back mutated.
Every switch degrades both. It is the same event as a PR job running against a
shared cache, and it wants the same answer.
One lineage per branch, with a fallback
Reads try the current scope, then each fallback in order. Writes only ever go to
scope— never to a fallback, not even the one just seeded from. That asymmetryis the whole of the isolation: a PR build cannot advance the cache its base builds
from, and a broken experiment on a branch cannot corrupt the one you go back to.
A new lineage is seeded from its fallback rather than starting cold, so a
branch switch costs a copy instead of a rebuild.
${git:branch}Expanded by the engine, reading
.git/HEADdirectly rather than shelling out —this runs on every engine construction, and a subprocess for one line of a file
that is always there is not worth it, nor is requiring
giton PATH.Outside a checkout, or on a detached HEAD, it resolves to the empty scope: one
shared lineage, today's behaviour. A cache policy must never be the reason a build
cannot start.
Scopes are sanitized to a single path component, because branch names routinely
contain
/and a raw one would nest an extra level and make two branches collidethe moment one is a path prefix of another.
Seeding is the only copy in the design
Once per (slot, scope), amortized over every later build on that branch, and
measured against a cold rebuild rather than against nothing.
It defaults on, because scoping without it just makes every branch switch cold —
worse than not scoping at all.
seedOnFork: falseturns it off for a large slot ona filesystem with no reflink. A reflink would make this near-free on APFS/btrfs and
is the obvious next step, but it is a per-platform optimization and this has to be
correct everywhere first.
A failed seed is a cold cache and never a failed build, per the scratch contract,
and leaves no partial tree behind for the next run to mistake for a warm one.
copy_treerecreates symlinks rather than following them, so a slot that hasacquired a link to somewhere else does not duplicate that somewhere else into the
new scope.
Compatibility
Defaults are unchanged.
scopeis empty, so a workspace that configuresnothing keeps the single lineage it has today, in the same directory layout. No
proto change, no ABI bump.
One live question for you: should
scopedefault to${git:branch}? On iswhat makes switching branches worth anything; it also multiplies a developer's
cache by branch count and makes the seed copy a routine cost. Left off here
because it is a policy call, not an implementation one.
Tests
17 unit + 2 e2e. The e2e pair is the story end to end — build on
master, switchto
featand seemaster's work, switch back and confirm the branch's writesnever reached
master— plus theseedOnFork: falsepath.Stack created with GitHub Stacks CLI • Give Feedback 💬