✨ feat: add a deterministic contextual <Glob> component - #229
Merged
Conversation
A document decides what to work on by looking at what is there. `<Glob>` answers that question against the contextual `Env.cwd`, so it composes with `<TempDir>` and `<File>` without any of them knowing about the others — a path it returns is a path `<File>` can read. It is a value component: `include` is required, `exclude` optional, and what it binds is a *set* of relative POSIX paths, deduplicated and sorted by code point. None of that comes from the order the filesystem handed entries back, because a document that branches on a listing must branch the same way on every host. Finding nothing is a result, not a failure. Only regular files come back. A symlink is a link rather than a file, so it is never a result and a link to a directory is never descended into — which is also what keeps traversal inside `Env.cwd` and free of cycles, without judging any destination. A pattern that cannot match anything a relative search produces — absolute, a leading `..`, or empty — fails rather than quietly contributing nothing: an empty result has to keep meaning "there are no such files". Two defects in `API.Fs.glob` had to be fixed for any of that to hold: - `exclude` was passed to `walk`'s `skip`, which tests patterns against the absolute entry path, so `.git/**` never matched anything and excluded nothing. Matching now happens on the relative POSIX path, and an exclusion covering a directory's contents prunes the subtree. - `walk` produces entries from a spawned task, so a `readdir` that failed tore down the surrounding scope as an uncaught rejection instead of throwing at the call site — no caller could report it. Traversal is now a plain recursive generator, which also makes every directory read a cancellation point. `resolveTestTarget` is the only other caller; it passes no `exclude` and filters on `isFile`, so its behavior is unchanged. The errno-to-phrase allowlist moves out of `File.ts` into `components/fs-diagnostics.ts`, shared by both filesystem components: a platform error names the path it failed on, and for a search that is a path the document never wrote. Closes #191
PR #229: ✨ feat: add a deterministic contextual component10 files, +1899 / -93 Scope🔴 PR has 1992 lines changed. Split into focused PRs. 🟡 1992 lines changed. PRs under 400 receive more thorough review. Structural✅ No structural bloat detected. Slop✅ Slop indicators look low. Static Analysis✅ Oxlint found no issues. CorrectnessNo extraneous code patterns detected. |
Exclusion pruning was unsound in two ways, and both dropped files the
exclusion never named.
A directory whose own path matched an exclusion was skipped along with
everything under it. But a directory is not a candidate — it is never
reported — so matching it says nothing about the paths beneath it:
`exclude: ["vendor"]` returned nothing at all instead of every file under
`vendor`.
Matching `<directory>/` was then treated as proof that every descendant was
excluded. That is false for any pattern whose last segment stops at a
separator, because `*` may match zero characters but cannot cross one. With
foo/direct.md
foo/deep/keep.md
include: ["**/*.md"]
exclude: ["foo/*"]
`foo/*` matches `foo/`, so `foo` was pruned and the result was `[]` — where
the exclusion selects only `foo/direct.md` and `foo/deep/keep.md` should
survive.
Exclusion is now decided per candidate, against that candidate's own relative
path, and a directory's path is not tested against exclusions at all. A subtree
is skipped only when a pattern provably covers every descendant, which is a
trailing `/**` — it compiles to `(?:[^/]*(?:/|$))*`, so once the part before it
matches a directory the pattern matches every path underneath at any depth.
`**` alone covers the whole tree the same way. Every other exclusion walks the
subtree and filters its files individually, which is the conservative
direction: descending needlessly costs reads, while skipping wrongly loses a
match.
Pruning is now an optimization that cannot change the answer. `**/*` excludes
every file at any depth and earns no pruning, and the result is the same either
way.
Tier GA is new and drives `glob()` directly, so no component runs and nothing
a component could do afterwards can satisfy it. It separates filtering from
pruning by counting directory reads — a result set alone cannot tell a
correctly filtered walk from an over-eager prune, because the two can agree on
the answer and disagree on the work. `.git/**` and `**/node_modules/**` are
asserted to read nothing below them.
`Glob.test.md` gains the two contract-level regressions: `vendor/*` keeps
`vendor/deep/keep.md`, and `vendor` removes nothing.
The specification and website documented the unsound rule and now describe the
sound one.
| } | ||
|
|
||
| // Code point order, not `localeCompare`: what a document branches on must not | ||
| // depend on the locale the host happens to be configured with. |
There was a problem hiding this comment.
Redundant comment — restates what the code does.
Suggested change
| // depend on the locale the host happens to be configured with. |
| } | ||
|
|
||
| // A symlink is reported by its own path and never followed, so traversal | ||
| // stays under the root and cannot cycle. |
There was a problem hiding this comment.
Redundant comment — restates what the code does.
Suggested change
| // stays under the root and cannot cycle. |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A document decides what to work on by looking at what is there — repository
instructions, test fixtures, source files, workflow inputs. Until now it had to
shell out to
findorlsand parse the result, which is neither deterministicnor something an author can assert against.
Closes #191. Follows #176 (schema-validated returns), #189 (
<TempDir>), and#190 (
<File>).What changes
<Glob>is a new built-in value component.Before:
— output that varies by host, by locale, and by directory order, with no way to
branch on it.
After:
<Glob include={["**/AGENTS.md"]} as="instructionPaths" />— a
string[]of relative POSIX paths, deduplicated and sorted by code point.Because patterns are relative to the contextual
Env.cwd, the three filesystemcomponents become a pipeline without any of them knowing about the others — a
path
<Glob>returns is a path<File>can read:How it works
Patterns are checked before the filesystem is touched,
Env.cwdis checkedbefore traversal, and matching itself belongs to the Fs Api — the component adds
no syntax of its own.
Review guide
Start with:
packages/core/src/components/Glob.test.md— the public matchingcontract, entirely in Markdown.
Then review:
specs/executable-mdx-spec.md§6.14 — props, dialect, ordering,normalization, symlinks, failures
packages/core/src/components/Glob.ts— the componentpackages/runtime/apis.ts— the twoAPI.Fs.globfixes belowpackages/core/tests/glob-component.test.ts— Tier GBpackages/core/src/components/fs-diagnostics.ts+File.ts— the extractederrno allowlist
site/routes/docs/components.tsx— website sectionLook carefully at: the traversal in
descend(). It is where containment,cancellability, and exclusion pruning all live.
Two defects in
API.Fs.globNeither of the issue's acceptance criteria for
excludeor traversal failurecould hold without fixing these, so they are in scope.
excludenever matched anything. It was passed to@effectionx/fs'swalk({ skip }), which tests patterns against the absolute entry path — soglobToRegExp(".git/**"), anchored^\.git\/…$, could never match/tmp/x/.git/config. Matching now happens on the relative POSIX path.A traversal failure was an uncaught rejection.
walkproduces entries froma spawned task, so a
readdirthat threw tore down the surrounding scopeinstead of throwing at the call site — no caller could catch or report it.
Verified before the change: a
chmod 000subdirectory crashed the process withUncaught (in promise) Error: EACCESpast atry/catcharound the call.Traversal is now a plain recursive generator, which also makes every directory
read a cancellation point.
What must stay true
Env.cwd— enforced by never following a symlink(a link is not a directory to descend, and not a file to return), checked by
GB15/GB15b/GB16/GB16b.
timing out rather than hanging.
GB19 and by the ordering/dedup tests in
Glob.test.md.and the errno selects an allowlisted phrase; checked by GB12, GB12b, GB13,
and four adversarial GB14 cases including an externally thrown
GlobError.anything fails instead, checked by GB6/GB7/GB8.
trailing
/**provably covers every descendant; every other exclusion walksand filters per file. Checked by Tier GA, which counts directory reads because
a result set cannot tell a correct filter from an over-eager prune.
How to verify it
Glob.test.md(27 assertions) proves the matching contract with noJavaScript, and fails if include/exclude precedence, ordering, dedup,
normalization, hidden-file semantics, or files-only filtering regress.
GB9proves an uncompilable pattern is reported as an authoring error andfails if a raw
RegExpmessage ("Unterminated character class") leaks.GB7proves refusing only a leading..is sufficient: a..furtheralong (
docs/../../../etc/passwd) matches nothing because traversal neverproduces such a path.
GB17proves traversal is cancellable mid-walk — the second directory readsuspends, the third is never read, and the document gets nothing.
GB18/GB18bprove replay: a completed root restores the array withoutsearching; a partial journal searches again and sees newly added files.
GA2/GA2b/GA3prove exclusions are read as written:foo/*keepsfoo/deep/keep.mdand still walksfoo, andvendoralone removes nothing.They fail if a directory match is ever taken as proof about its descendants.
GA4/GA4b/GA5prove pruning is sound and still happens:.git/**and**/node_modules/**read nothing below them, while the equally-total**/*walks and filters — same answer, different work.
Verified on Deno 2.9.1 (CI-pinned):
lint,check,test(215 passed /1723 steps),
check:jsrall clean. Alsotest:node(1698/0),test:bun(1698/0),
sitecheck, and./dist/xmd test packages/core/srcfrom thecompiled binary.
Scope
Included
<Glob>, registered as a built-inAPI.Fs.globfixes: relative-path exclusion matching, catchabletraversal failures, and sound subtree pruning
glob()directly rather than through any callerFile.tsIntentionally unchanged
resolveTestTargetis the only otherglobcaller. It passes noexcludeand filters on
isFile, so its behavior is identical — confirmed by thetest-targetsuite and by the compiled binary still discovering everydocument under
packages/core/src.API.Fs.globkeeps returning{ path, isFile }and keeps reporting symlinkswith
isFile: false. Baking "files only" into the Api would be thecomponent's opinion, not the Api's.
@effectionx/fsexposesfollowSymlinksbut guarantees neither root containment nor cycle detection.New abstractions
components/fs-diagnostics.tsexists because the errno-to-phrase allowlist isnow needed by two components with the same constraint; duplicating fourteen
entries would let them drift.
GlobErrorexists to mark the component's own failures, on the same terms asFileAccessError.Risks and limitations
<Glob>never follows a symlink, so nothing it reads is chosenby one — but a directory that is real when read could be replaced afterwards.
Containment independent of observed filesystem state is Contain filesystem access at the provider boundary #227.
/**earns it. Apattern that happens to cover a whole subtree by other means — a character
class enumerating every child, say — is walked and filtered instead. That
costs reads and never costs correctness, which is the direction to be wrong
in.
Scope confirmation