Skip to content

feat(cache): classify auto-tracked inputs and outputs from access order#571

Draft
wan9chi wants to merge 21 commits into
mainfrom
feat/auto-tracking-read-write-overlap
Draft

feat(cache): classify auto-tracked inputs and outputs from access order#571
wan9chi wants to merge 21 commits into
mainfrom
feat/auto-tracking-read-write-overlap

Conversation

@wan9chi

@wan9chi wan9chi commented Jul 26, 2026

Copy link
Copy Markdown
Member

Motivation

Automatic input/output tracking cannot currently classify a path that a task
both reads and writes, and it has no notion of an output directory being cleaned
and republished. Two consequences show up on real workloads:

  • Atomic publishing looks like nothing happened. A task that stages into
    dist.tmp and renames the directory into place has all of its writes recorded
    against a path that no longer exists at archive time. The archive comes out
    empty, so a later cache hit restores nothing and leaves a wrong tree.
  • A task that reads its own derived state never settles. Generated files
    under a gitignored output directory get recorded as inputs, so the fingerprint
    changes every run and the task can never hit.

Together these are why a workspace like emdash
(20 packages, mostly plain tsdown) could not cache its build without
hand-written input/output globs. With this change it caches with no cache
configuration at all.

Approach

Classification is driven by the order of accesses to each path plus whether
the path is gitignored:

  • write-only, or written before it was read → output
  • read before it was written → output if gitignored, otherwise a modified input
  • a rename destination counts as a write, and renaming a directory
    re-attributes the writes recorded under the old subtree
  • an output that does not exist at archive time is dropped, but only if its
    absence is explained by an ancestor rename; otherwise the task is not cached
  • a gitignored path underneath a directory this task wrote into is the task's
    own derived state, and is not an input

The last two matter for safety: a missed signal must degrade into a missed cache
entry, never a wrong tree.

To make this observable, fspy gains mutation intent. AccessMode widens to
u16 and adds CREATE, TRUNCATE, EXCLUSIVE, RENAME_FROM, RENAME_TO,
DELETED and IS_DIR. Rename and delete are now intercepted on all three
backends — on Windows both travel through NtSetInformationFile rather than
dedicated syscalls. CreateDisposition / open flags supply create and truncate,
which is what distinguishes a handle merely opened for writing from one that
actually replaces content.

Every flag is derived from a call's arguments and never from its result. The
Linux seccomp supervisor responds with SECCOMP_USER_NOTIF_FLAG_CONTINUE, so it
is notified before the syscall runs and can never learn the outcome. An
outcome-dependent rule would silently hold on macOS and Windows but not Linux.

CACHE_SCHEMA_VERSION goes 18 → 19, so existing entries are invalidated.

Notes for review

  • DECISIONS.md records the rationale and the points where the design changed
    during development, including which earlier decisions the
    arguments-not-outcomes constraint invalidated. Happy to drop it from the
    branch if it is not wanted in the repo.
  • Two commits fix pre-existing Windows-only clippy failures (vite_powershell,
    fspy::windows). Both items are behind cfg(windows), so the native lint runs
    never reach them and just lint-windows is red on main without them. They
    are unrelated to tracking.
  • One pre-existing bug is not addressed here: pnpm's SEA build aborts with
    Assertion failed: (magic) == (kMagic) when run under fspy. It reproduces on
    main and blocks one emdash package.
  • INVESTIGATE.md records two tracking issues these rules do not fix, both
    found while verifying against emdash. --parallel reproducibly loses cache hits
    because a reader fingerprints a dependency's output directory mid-write, and
    three emdash tasks never hit. Both trace to pnpm self-linking a workspace
    package, so packages/core/dist and node_modules/emdash/dist are one
    directory under two names and the own-derived-state rule only recognises the
    first. Neither is a regression; the second is a case these rules were meant to
    cover but attribute by the wrong path.

wan9chi and others added 15 commits July 26, 2026 22:42
Extend AccessMode to u16 and add FAILED, CREATE, TRUNCATE, EXCLUSIVE,
RENAME_FROM, RENAME_TO, DELETED, CREATED_DIR and IS_DIR.

Interceptions in the Unix preload now report after forwarding to the real
function so the outcome is known, and rename, unlink, rmdir and mkdir are
intercepted for the first time.

Co-authored-by: Claude <noreply@anthropic.com>
…itignore

Decide from ordered accesses which paths a task consumed and which it
produced, using only information available from a call's arguments.

A write with no read is an output. A write-first overlap is an output. A
read-first overlap is an output when the path is gitignored and a modified
input otherwise, which is what separates a linter rewriting its own cache
from a formatter rewriting a source. Rename destinations count as writes and
directory renames re-attribute the writes recorded beneath them, so a build
that stages into dist.tmp and swaps it into place still reports its outputs.

Bare directory stats and listings of directories the task wrote into no
longer become inputs; such a fingerprint records the task's own product and
can never match on a clean checkout. PathFingerprint::Folder therefore always
carries entries, so the cache schema moves to v19.

Co-authored-by: Claude <noreply@anthropic.com>
Renaming a staging directory takes everything beneath it along, but only the
directory itself carries the rename event. Without checking ancestors, every
file written under dist.tmp looked like a mutation that vanished unexplained,
which blocked caching for the atomic-publish pattern the rename support was
added to handle.

Co-authored-by: Claude <noreply@anthropic.com>
touch-file opens O_RDWR and never writes a byte, which leaves the file
untouched. Treating write capability as a mutation is the false positive this
rule set removes: Biome opens a clean source read-write on every run, and lock
files across cargo, rustc and Parcel are opened O_RDWR and only flocked.

rw-pkg, which genuinely reads and rewrites a tracked source, still reports
that it modified its input.

Co-authored-by: Claude <noreply@anthropic.com>
Three cases for the auto-tracking rules, each asserting the outcome that
matters rather than just a cache hit.

atomic-pkg stages under dist.tmp and renames the directory into place, then
removes dist and runs again to prove the outputs came back from the archive.
clean-pkg empties its output directory every run, which is what makes a build
tool stat and list its own product. state-pkg reads and rewrites a gitignored
file, the cache half of a command like eslint --fix --cache.

The work happens inside one vtt process because a compound 'a && b' command is
cached as separate tasks, which would hide the relationship under test.

Co-authored-by: Claude <noreply@anthropic.com>
Satisfies clippy line limits and makes the two phases legible: folding the
event stream into per-path histories, then deriving the directory subtrees the
task wrote into.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
emdash's admin build missed on every run. Its build is a compound command whose
segments are separate tasks, and the tsdown segment reads
dist/locales/<locale>/messages.mjs while a later segment rewrites those files,
so the fingerprint could never settle.

Neither existing rule could reject it. The task did not write into
dist/locales itself, a sibling did, so the wrote-into-subtree rule stays quiet.
Rejecting purely on gitignore is unsafe because a package reading
node_modules/<dep>/dist is also reading ignored derived content, and that must
remain an input or changing a dependency stops invalidating its consumers.

A read now counts as the task's own derived state only when version control
calls it derived and it sits under a directory this same task wrote into.
Directory listings apply the same test.

emdash's 23 package builds now cache 23/23 with no manual globs, and deleting
every dist restores all 1358 files byte-for-byte.

Co-authored-by: Claude <noreply@anthropic.com>
Reproduces emdash's admin build: a compound command whose middle segment reads
a file inside its own output directory while a later segment writes there. Each
&& segment is a separate cached task, so this is the shape that used to miss on
every run.

Co-authored-by: Claude <noreply@anthropic.com>
Rename and delete reach the kernel through NtSetInformationFile on
Windows rather than through dedicated syscalls, so a single detour on
that entry point covers what rename, unlink and rmdir cover on Unix.
Without it a tool that publishes atomically is invisible: the write
lands on a temporary path that no longer exists at archive time, and
the destination is never seen to change.

NtCreateFile now also folds CreateDisposition into the access mode.
Disposition is Windows' O_CREAT/O_TRUNC/O_EXCL, and without it a handle
opened for writing cannot be told apart from one that actually replaces
content -- only the latter is a mutation.

Both interceptions record the call's arguments and never its result,
matching the Unix backends. The Linux seccomp supervisor is notified
before the syscall runs and can never observe outcomes, so an
outcome-dependent rule would hold on some platforms and not others.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Both items are gated behind cfg(windows), so the native macOS and Linux
clippy runs never reach them and these failures only surface under
`just lint-windows`. Neither is related to file-tracking work; the
cross-target lint gate is red on main without them.

`unused_async_trait_impl` is a newer sibling of `unused_async`, which
FspySpawner::spawn already expects for the same reason: the async
signature is required by the SpyImpl trait.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Pure formatting from `just fmt` (rustfmt wrapping, module ordering,
prettier on the new fixtures and DECISIONS.md). No behaviour change.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@wan9chi wan9chi changed the title feat(fspy): record access outcome and mutation intent feat(cache): classify auto-tracked inputs and outputs from access order Jul 26, 2026

wan9chi commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@socket-security

socket-security Bot commented Jul 26, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​ignore@​0.4.319810093100100

View full report

wan9chi and others added 3 commits July 26, 2026 22:56
Vitest reads and then rewrites `node_modules/.vite/vitest/*/results.json`
on every run. Resolving that read-first overlap through gitignore alone
called it a modified input, so the task could never cache: the browser
fixture reported "not cached because it modified its input" on all three
runs.

The gitignore proxy assumed a workspace has a `.gitignore` covering its
derived state. The fixture has none, and a package or a non-repository
workspace need not either. `node_modules` is installed rather than
authored and `.git` is the repository itself, so neither is ever an
authored source and neither needs a rule to say so.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`assert_contains` compared the accumulated mode for exact equality, so
adding CREATE and TRUNCATE broke every write assertion in node_fs.

Spelling the new flags out per test would assert the runtime's open flags
rather than the behaviour under test, and they genuinely differ:
`writeFileSync` truncates on Node and Deno but not on Bun. These modifiers
now only participate when a test names one explicitly, leaving the read
versus write distinction as strict as before.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
musl builds no preload library and rely entirely on seccomp, which only
watched open, stat, getdents and execve. A task that publishes its output
by renaming a staging directory into place therefore looked like it wrote
nothing that still existed, and the atomic-publish case reported "not
cached because it modified its input" on musl while passing everywhere
else.

Source and destination both come from the call's arguments, so this adds
no outcome dependency: the supervisor still responds with
SECCOMP_USER_NOTIF_FLAG_CONTINUE and never learns whether the call
succeeded. Whether the source is a directory is read from the filesystem
before the syscall runs, which is existing state rather than a result, and
it decides whether writes under the old subtree are re-attributed.

renameat2 needs a fifth argument, so FromNotify gains a 5-tuple.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
wan9chi added a commit to voidzero-dev/vite-plus that referenced this pull request Jul 26, 2026
Picks up voidzero-dev/vite-task#571, which classifies automatically
tracked paths that a task both reads and writes.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
wan9chi added a commit to wan9chi/emdash that referenced this pull request Jul 26, 2026
TEMPORARY, not for merge. Pins vite-plus to the preview build of
voidzero-dev/vite-plus#2260, which carries the automatic tracking change
in voidzero-dev/vite-task#571, so CI exercises it on this workspace.

The bridge registry is what makes the same version resolve in CI, so it
goes in .npmrc rather than being passed on the command line. Bridge builds
carry no provenance attestation and are published minutes before use, so
they need exemptions from both the trust policy and the release-age
cooldown.

Revert all four pieces to return to the published release: the .npmrc
registry, the version pin, and the two pnpm-workspace exclusions.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
wan9chi and others added 2 commits July 27, 2026 09:54
Neither is a regression from this branch, and neither is fixed here.

The `--parallel` hit-rate loss and the three tasks that never hit both
trace to pnpm self-linking a workspace package, so `packages/core/dist`
and `node_modules/emdash/dist` are one directory under two names. The
own-derived-state rule attributes by path and only recognises the first.

Records what is measured, what is still hypothesis, and the reproduction
for each, including one hypothesis already ruled out: the files appearing
mid-run are core's own output, not a second writer.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Rewrites the investigation notes now that both causes are measured rather
than suspected, and corrects the earlier `--parallel` entry.

Two causes, both fixable here. A workspace package reachable under two
names via its node_modules self-link, so the own-derived-state rule from
#571 compares path strings and misses its own output. And a package
manager's workspace-state file, which carries a wall-clock timestamp and
absolute paths, so any task wrapping `pnpm run` is uncacheable across
machines and also defeats cache portability across workspace roots.

`--parallel` was recorded as a possible correctness bug. It is documented
as running without dependency ordering, and timestamped task starts confirm
it behaves as documented, so that entry is withdrawn.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant