fields: precompile templates into reusable segments - #43
Merged
Conversation
Introduce fields::Template: Compile() parses a template once into literal
and field segments; Render() walks the segments per entry, appending each
literal verbatim and resolving each field. The free Render(tmpl, ...) now
delegates to Compile().Render(), so existing call sites are unchanged.
run.cc compiles the --template once before the walk and renders each match
through the compiled form, replacing the per-entry re-scan of the template
string. This is the prerequisite for moving field-name resolution to compile
time (constexpr dispatch) and for threading {root} via a render context.
No behaviour change; adds a CompiledTemplate reuse test.
helly25
added a commit
that referenced
this pull request
Jun 24, 2026
Add --sort[=name], an xff extension: order each directory's entries by name before
visiting, so the walk is reproducible and diffable instead of filesystem-readdir
order (find's default, which --sort=none / absence preserves). Bare --sort means
--sort=name; last occurrence wins.
- WalkOptions gains SortOrder { kNone, kName }; Walker::Descend sorts the ReadDir
result by path (siblings share the dir prefix, so path order is name order) when
kName, covering both pre- and post-order.
- run.cc ResolveSort(globals) maps --sort / --sort=name -> kName, --sort=none -> kNone.
Tests: run_test (--sort visits root, a.txt < b.md < sub, sub/c.txt in that exact
order via ElementsAre; the unsorted default stays UnorderedElementsAre). Green in
both configs; clang-format + em-dash clean.
The parallel-traversal half of #43 remains a separate follow-up.
helly25
added a commit
that referenced
this pull request
Jun 27, 2026
Design of record for parallel directory traversal and --sort, settled with the user: worker pool over directories + one emission-ordering layer owning the sink; four --sort modes (none/dir/subtree/tree) as a rising-cost spectrum; single -j/--jobs knob for walk + exec; mode-scoped defaults (find/fd/rg = all cores + none; modern = max(1,min(cores-1,15)) + dir); concurrency correctness for prune/quit/depth/exit-code; a separate --config=tsan + clang-tsan CI cell landed with the threads; phased as a chain of stacked PRs. No code yet -- this is the spec to build to.
helly25
added a commit
that referenced
this pull request
Jun 27, 2026
* engine: parallel directory traversal + --sort modes + -j (#43) Add a bounded worker pool (ReadPool) that runs readdir+lstat off the coordinator thread; the visitor still runs single-threaded in --sort order, so evaluate/emit/exec/capture/summary stay unchanged and race-free. Workers are pure (path -> stat'd listing), touching only the thread-safe VFS and a mutex-guarded queue. --sort grows to none|dir|subtree|tree (name aliases dir): none = readdir order; dir = each directory's sorted listing block then its subtrees; subtree = sorted non-dir entries then contiguous subtrees; tree = total path order. The ordered modes are deterministic (siblings consumed in sorted order while the pool prefetches their reads). -j N / --jobs=N sets the worker count (default 1 = sequential); the parent batches subdirectory reads so the pool overlaps their IO. Adds --config=tsan and a ubuntu-only, repo-cache-only clang-tsan CI cell (kept off the disk cache so the 10 GB Actions budget stays healthy next to asan). walk_test covers parallel set-equality and deterministic ordering for all modes at workers=1 and 4; full suite is green under asan+ubsan and tsan. Deferred (follow-ups): mode-scoped auto-defaults (modern -> parallel+dir) with the mode mechanism (#54); completion-order subtree streaming; per-worker eval. * docs: note v1 scope in parallel-traversal design Record the three deliberate v1 simplifications: IO-only parallelism with a single-threaded visitor, deterministic (sorted) ordered modes, and opt-in -j/--sort with mode-scoped auto-defaults deferred to the mode mechanism (#54).
helly25
added a commit
that referenced
this pull request
Jun 27, 2026
…unordered) (#144) * engine: mode-scoped traversal defaults (modern parallel+sorted, find unordered) Completes the deferred piece of the parallel walk (#43/#54). RunFind gains an optional registry::Style: when the user gives no --sort / -j, the default is mode-scoped - kXff (modern) sorts each directory (--sort=dir) and runs a capped parallel walk (max(1, min(cores-1, 15))); kFind matches find (unordered) but saturates cores. std::nullopt keeps the conservative unordered + sequential default, so the ~32 in-process RunFind callers and conformance are untouched; the CLI passes the active style (config::ActiveStyle / argv[0] dispatch). run_test::ModeScopedSortDefault locks the behavior (kXff deterministic dir- sorted; kFind same set, unordered). Full suite green under default and asan+ubsan; tsan unaffected (test callers stay sequential; walk_test already covers the parallel path). * test+style: use unqualified ElementsAre; document the no-::testing-prefix rule run_test's ModeScopedSortDefault wrote ::testing::ElementsAre inline even though the file already has `using ::testing::ElementsAre;` - use the bare name. STYLE_CPP.md now states the rule explicitly: bring matchers in with a using and never write the ::testing:: prefix inside an EXPECT_THAT/ASSERT_THAT expression (fixture utilities like ::testing::Test are exempt). Audited the suite: this was the only inline-qualified matcher.
helly25
added a commit
that referenced
this pull request
Jun 28, 2026
…oadmap decisions (#193) Per the directive to put enforcement rules in the repo, not just memory: AGENTS.md gains two sections: - Self-documenting features: the registry + globals are the doc SOT; every feature add/change updates its Descriptor.summary / GlobalFlag entry + kHelpText in the same change, so --help/--man/--markdown stay complete. - CLI conventions: flag scope by dash count (--global vs -primary; -h/-q/-help/ -version are special-cased compat globals); flag-only, no subcommands; and a user-toggleable boolean capability is a --feature, not a one-off flag. TODO.md roadmap decisions (design phase, accounting for existing code): - #43 parallel traversal + --sort: already BUILT (ReadPool, sort modes, tests, tsan cell); remaining is a CLI bashtest. - #45 --exact/--path-encoding: default = filesystem-native (natural per-platform case behavior); --exact forces verbatim byte matching; --path-encoding=raw|escape. - #73 --feature: PARKED (no customer yet) with the full ready-to-build design + a trigger (first boolean capability builds it); trigger mirrored in AGENTS.md. - #54 mode mechanism: subsumed by --config (no --mode flag; --modern deferred).
helly25
added a commit
that referenced
this pull request
Jul 6, 2026
…280) * docs(TODO): give #109, smart-case, and #84 their own TODO.md entries These three were tracked only in the session task list / memory with no TODO.md home. Add roadmap-tail bullets so TODO.md is complete: - Hash-verification workflow (#109) [DISCUSS] - -eval matcher + read-into-var + --summary tallies; the hashing primitives already shipped (#105). - Smart-case matching (--smart-case) [DISCUSS] - rg/fd convention; sequence before the flavor feature-map help row. - Sharded-file support (#84) TBD - needs a design pass. Fold in a correction: the old 'Per-file content hashes' bullet was stale - the {hash} field + -hash action shipped in #105 (fields.cc, registry.cc); its only still-open aspect (verify/match) is now the #109 entry. * docs(TODO): promote the --sort/-j CLI bashtest to its own bullet It was a sub-clause buried in the (shipped) #43 parallel-traversal bullet, so it had no findable standalone location. Give it its own roadmap-tail bullet and point the #43 record at it, so every tracked item maps to its own line.
helly25
added a commit
that referenced
this pull request
Jul 7, 2026
sort_test.sh drives the real binary over a fixed tree: every --sort mode (none/dir/subtree/tree) walks the whole tree; --sort=tree is a deterministic global order that is byte-identical across worker counts (-j1 vs -j8, the core parallel-determinism guarantee); --sort=dir orders each directory's entries; and --jobs=all / -j N visit everything. The engine unit tests already cover ordering across worker counts + a tsan cell; this adds the missing CLI-level coverage. TODO.md: #43/#27 marked complete.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Precompiles named-field templates so the hot path (rendering every match) stops re-scanning the template string per entry (#51).
fields::Template—Compile()parses a template once into literal/field segments;Render(path, metadata, depth)walks the segments, appending literals verbatim and resolving fields.Render(tmpl, ...)now delegates toCompile().Render(), so every existing call site and test is unchanged.run.cccompiles the--templateonce before the walk and renders each match through the compiled form (was:fields::Render(*tmpl, …)re-parsing the string on every matched entry).Why now
This is the architectural prerequisite for the remaining #51 slices:
Compile-time lookup;{root}through a render context instead of adding another positional parameter.No behaviour change.
Test plan
CompiledTemplate RendersManyEntries: oneTemplate::Compilereused across two entries, covering a plain field + literal + a qualified field ({size:h}).fields_testcases pass unchanged via the delegating freeRender.bazel test //...(default) and--config=clang --config=asan.