feat(cache): classify auto-tracked inputs and outputs from access order#571
Draft
wan9chi wants to merge 21 commits into
Draft
feat(cache): classify auto-tracked inputs and outputs from access order#571wan9chi wants to merge 21 commits into
wan9chi wants to merge 21 commits into
Conversation
Extend AccessMode to u16 and add FAILED, CREATE, TRUNCATE, EXCLUSIVE, RENAME_FROM, RENAME_TO, DELETED, CREATED_DIR and IS_DIR. Interceptions in the Unix preload now report after forwarding to the real function so the outcome is known, and rename, unlink, rmdir and mkdir are intercepted for the first time. Co-authored-by: Claude <noreply@anthropic.com>
…itignore Decide from ordered accesses which paths a task consumed and which it produced, using only information available from a call's arguments. A write with no read is an output. A write-first overlap is an output. A read-first overlap is an output when the path is gitignored and a modified input otherwise, which is what separates a linter rewriting its own cache from a formatter rewriting a source. Rename destinations count as writes and directory renames re-attribute the writes recorded beneath them, so a build that stages into dist.tmp and swaps it into place still reports its outputs. Bare directory stats and listings of directories the task wrote into no longer become inputs; such a fingerprint records the task's own product and can never match on a clean checkout. PathFingerprint::Folder therefore always carries entries, so the cache schema moves to v19. Co-authored-by: Claude <noreply@anthropic.com>
Renaming a staging directory takes everything beneath it along, but only the directory itself carries the rename event. Without checking ancestors, every file written under dist.tmp looked like a mutation that vanished unexplained, which blocked caching for the atomic-publish pattern the rename support was added to handle. Co-authored-by: Claude <noreply@anthropic.com>
touch-file opens O_RDWR and never writes a byte, which leaves the file untouched. Treating write capability as a mutation is the false positive this rule set removes: Biome opens a clean source read-write on every run, and lock files across cargo, rustc and Parcel are opened O_RDWR and only flocked. rw-pkg, which genuinely reads and rewrites a tracked source, still reports that it modified its input. Co-authored-by: Claude <noreply@anthropic.com>
Three cases for the auto-tracking rules, each asserting the outcome that matters rather than just a cache hit. atomic-pkg stages under dist.tmp and renames the directory into place, then removes dist and runs again to prove the outputs came back from the archive. clean-pkg empties its output directory every run, which is what makes a build tool stat and list its own product. state-pkg reads and rewrites a gitignored file, the cache half of a command like eslint --fix --cache. The work happens inside one vtt process because a compound 'a && b' command is cached as separate tasks, which would hide the relationship under test. Co-authored-by: Claude <noreply@anthropic.com>
Satisfies clippy line limits and makes the two phases legible: folding the event stream into per-path histories, then deriving the directory subtrees the task wrote into. Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
emdash's admin build missed on every run. Its build is a compound command whose segments are separate tasks, and the tsdown segment reads dist/locales/<locale>/messages.mjs while a later segment rewrites those files, so the fingerprint could never settle. Neither existing rule could reject it. The task did not write into dist/locales itself, a sibling did, so the wrote-into-subtree rule stays quiet. Rejecting purely on gitignore is unsafe because a package reading node_modules/<dep>/dist is also reading ignored derived content, and that must remain an input or changing a dependency stops invalidating its consumers. A read now counts as the task's own derived state only when version control calls it derived and it sits under a directory this same task wrote into. Directory listings apply the same test. emdash's 23 package builds now cache 23/23 with no manual globs, and deleting every dist restores all 1358 files byte-for-byte. Co-authored-by: Claude <noreply@anthropic.com>
Reproduces emdash's admin build: a compound command whose middle segment reads a file inside its own output directory while a later segment writes there. Each && segment is a separate cached task, so this is the shape that used to miss on every run. Co-authored-by: Claude <noreply@anthropic.com>
Rename and delete reach the kernel through NtSetInformationFile on Windows rather than through dedicated syscalls, so a single detour on that entry point covers what rename, unlink and rmdir cover on Unix. Without it a tool that publishes atomically is invisible: the write lands on a temporary path that no longer exists at archive time, and the destination is never seen to change. NtCreateFile now also folds CreateDisposition into the access mode. Disposition is Windows' O_CREAT/O_TRUNC/O_EXCL, and without it a handle opened for writing cannot be told apart from one that actually replaces content -- only the latter is a mutation. Both interceptions record the call's arguments and never its result, matching the Unix backends. The Linux seccomp supervisor is notified before the syscall runs and can never observe outcomes, so an outcome-dependent rule would hold on some platforms and not others. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Both items are gated behind cfg(windows), so the native macOS and Linux clippy runs never reach them and these failures only surface under `just lint-windows`. Neither is related to file-tracking work; the cross-target lint gate is red on main without them. `unused_async_trait_impl` is a newer sibling of `unused_async`, which FspySpawner::spawn already expects for the same reason: the async signature is required by the SpyImpl trait. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Pure formatting from `just fmt` (rustfmt wrapping, module ordering, prettier on the new fixtures and DECISIONS.md). No behaviour change. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Member
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Vitest reads and then rewrites `node_modules/.vite/vitest/*/results.json` on every run. Resolving that read-first overlap through gitignore alone called it a modified input, so the task could never cache: the browser fixture reported "not cached because it modified its input" on all three runs. The gitignore proxy assumed a workspace has a `.gitignore` covering its derived state. The fixture has none, and a package or a non-repository workspace need not either. `node_modules` is installed rather than authored and `.git` is the repository itself, so neither is ever an authored source and neither needs a rule to say so. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`assert_contains` compared the accumulated mode for exact equality, so adding CREATE and TRUNCATE broke every write assertion in node_fs. Spelling the new flags out per test would assert the runtime's open flags rather than the behaviour under test, and they genuinely differ: `writeFileSync` truncates on Node and Deno but not on Bun. These modifiers now only participate when a test names one explicitly, leaving the read versus write distinction as strict as before. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
musl builds no preload library and rely entirely on seccomp, which only watched open, stat, getdents and execve. A task that publishes its output by renaming a staging directory into place therefore looked like it wrote nothing that still existed, and the atomic-publish case reported "not cached because it modified its input" on musl while passing everywhere else. Source and destination both come from the call's arguments, so this adds no outcome dependency: the supervisor still responds with SECCOMP_USER_NOTIF_FLAG_CONTINUE and never learns whether the call succeeded. Whether the source is a directory is read from the filesystem before the syscall runs, which is existing state rather than a result, and it decides whether writes under the old subtree are re-attributed. renameat2 needs a fifth argument, so FromNotify gains a 5-tuple. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
wan9chi
added a commit
to voidzero-dev/vite-plus
that referenced
this pull request
Jul 26, 2026
Picks up voidzero-dev/vite-task#571, which classifies automatically tracked paths that a task both reads and writes. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
wan9chi
added a commit
to wan9chi/emdash
that referenced
this pull request
Jul 26, 2026
TEMPORARY, not for merge. Pins vite-plus to the preview build of voidzero-dev/vite-plus#2260, which carries the automatic tracking change in voidzero-dev/vite-task#571, so CI exercises it on this workspace. The bridge registry is what makes the same version resolve in CI, so it goes in .npmrc rather than being passed on the command line. Bridge builds carry no provenance attestation and are published minutes before use, so they need exemptions from both the trust policy and the release-age cooldown. Revert all four pieces to return to the published release: the .npmrc registry, the version pin, and the two pnpm-workspace exclusions. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Neither is a regression from this branch, and neither is fixed here. The `--parallel` hit-rate loss and the three tasks that never hit both trace to pnpm self-linking a workspace package, so `packages/core/dist` and `node_modules/emdash/dist` are one directory under two names. The own-derived-state rule attributes by path and only recognises the first. Records what is measured, what is still hypothesis, and the reproduction for each, including one hypothesis already ruled out: the files appearing mid-run are core's own output, not a second writer. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Rewrites the investigation notes now that both causes are measured rather than suspected, and corrects the earlier `--parallel` entry. Two causes, both fixable here. A workspace package reachable under two names via its node_modules self-link, so the own-derived-state rule from #571 compares path strings and misses its own output. And a package manager's workspace-state file, which carries a wall-clock timestamp and absolute paths, so any task wrapping `pnpm run` is uncacheable across machines and also defeats cache portability across workspace roots. `--parallel` was recorded as a possible correctness bug. It is documented as running without dependency ordering, and timestamped task starts confirm it behaves as documented, so that entry is withdrawn. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Motivation
Automatic input/output tracking cannot currently classify a path that a task
both reads and writes, and it has no notion of an output directory being cleaned
and republished. Two consequences show up on real workloads:
dist.tmpand renames the directory into place has all of its writes recordedagainst a path that no longer exists at archive time. The archive comes out
empty, so a later cache hit restores nothing and leaves a wrong tree.
under a gitignored output directory get recorded as inputs, so the fingerprint
changes every run and the task can never hit.
Together these are why a workspace like emdash
(20 packages, mostly plain
tsdown) could not cache itsbuildwithouthand-written
input/outputglobs. With this change it caches with no cacheconfiguration at all.
Approach
Classification is driven by the order of accesses to each path plus whether
the path is gitignored:
re-attributes the writes recorded under the old subtree
absence is explained by an ancestor rename; otherwise the task is not cached
own derived state, and is not an input
The last two matter for safety: a missed signal must degrade into a missed cache
entry, never a wrong tree.
To make this observable,
fspygains mutation intent.AccessModewidens tou16and addsCREATE,TRUNCATE,EXCLUSIVE,RENAME_FROM,RENAME_TO,DELETEDandIS_DIR. Rename and delete are now intercepted on all threebackends — on Windows both travel through
NtSetInformationFilerather thandedicated syscalls.
CreateDisposition/ open flags supply create and truncate,which is what distinguishes a handle merely opened for writing from one that
actually replaces content.
Every flag is derived from a call's arguments and never from its result. The
Linux seccomp supervisor responds with
SECCOMP_USER_NOTIF_FLAG_CONTINUE, so itis notified before the syscall runs and can never learn the outcome. An
outcome-dependent rule would silently hold on macOS and Windows but not Linux.
CACHE_SCHEMA_VERSIONgoes 18 → 19, so existing entries are invalidated.Notes for review
DECISIONS.mdrecords the rationale and the points where the design changedduring development, including which earlier decisions the
arguments-not-outcomes constraint invalidated. Happy to drop it from the
branch if it is not wanted in the repo.
vite_powershell,fspy::windows). Both items are behindcfg(windows), so the native lint runsnever reach them and
just lint-windowsis red onmainwithout them. Theyare unrelated to tracking.
pnpm's SEA build aborts withAssertion failed: (magic) == (kMagic)when run underfspy. It reproduces onmainand blocks one emdash package.INVESTIGATE.mdrecords two tracking issues these rules do not fix, bothfound while verifying against emdash.
--parallelreproducibly loses cache hitsbecause a reader fingerprints a dependency's output directory mid-write, and
three emdash tasks never hit. Both trace to pnpm self-linking a workspace
package, so
packages/core/distandnode_modules/emdash/distare onedirectory under two names and the own-derived-state rule only recognises the
first. Neither is a regression; the second is a case these rules were meant to
cover but attribute by the wrong path.