Skip to content

fields: precompile templates into reusable segments - #43

Merged
helly25 merged 1 commit into
mainfrom
feat/field-precompile
Jun 20, 2026
Merged

fields: precompile templates into reusable segments#43
helly25 merged 1 commit into
mainfrom
feat/field-precompile

Conversation

@helly25

@helly25 helly25 commented Jun 20, 2026

Copy link
Copy Markdown
Owner

Summary

Precompiles named-field templates so the hot path (rendering every match) stops re-scanning the template string per entry (#51).

  • fields::TemplateCompile() parses a template once into literal/field segments; Render(path, metadata, depth) walks the segments, appending literals verbatim and resolving fields.
  • The free Render(tmpl, ...) now delegates to Compile().Render(), so every existing call site and test is unchanged.
  • run.cc compiles the --template once before the walk and renders each match through the compiled form (was: fields::Render(*tmpl, …) re-parsing the string on every matched entry).

Why now

This is the architectural prerequisite for the remaining #51 slices:

  • moving field-name resolution to compile time (the constexpr/mbo dispatch table) — the segment already carries a resolved field, so that becomes a Compile-time lookup;
  • threading {root} through a render context instead of adding another positional parameter.

No behaviour change.

Test plan

  • New CompiledTemplate RendersManyEntries: one Template::Compile reused across two entries, covering a plain field + literal + a qualified field ({size:h}).
  • All existing fields_test cases pass unchanged via the delegating free Render.
  • Green on both toolchains: bazel test //... (default) and --config=clang --config=asan.

Introduce fields::Template: Compile() parses a template once into literal
and field segments; Render() walks the segments per entry, appending each
literal verbatim and resolving each field. The free Render(tmpl, ...) now
delegates to Compile().Render(), so existing call sites are unchanged.

run.cc compiles the --template once before the walk and renders each match
through the compiled form, replacing the per-entry re-scan of the template
string. This is the prerequisite for moving field-name resolution to compile
time (constexpr dispatch) and for threading {root} via a render context.

No behaviour change; adds a CompiledTemplate reuse test.
@helly25
helly25 merged commit 2d9a62b into main Jun 20, 2026
4 checks passed
@helly25
helly25 deleted the feat/field-precompile branch June 20, 2026 17:01
helly25 added a commit that referenced this pull request Jun 24, 2026
Add --sort[=name], an xff extension: order each directory's entries by name before
visiting, so the walk is reproducible and diffable instead of filesystem-readdir
order (find's default, which --sort=none / absence preserves). Bare --sort means
--sort=name; last occurrence wins.

- WalkOptions gains SortOrder { kNone, kName }; Walker::Descend sorts the ReadDir
  result by path (siblings share the dir prefix, so path order is name order) when
  kName, covering both pre- and post-order.
- run.cc ResolveSort(globals) maps --sort / --sort=name -> kName, --sort=none -> kNone.

Tests: run_test (--sort visits root, a.txt < b.md < sub, sub/c.txt in that exact
order via ElementsAre; the unsorted default stays UnorderedElementsAre). Green in
both configs; clang-format + em-dash clean.

The parallel-traversal half of #43 remains a separate follow-up.
helly25 added a commit that referenced this pull request Jun 27, 2026
Design of record for parallel directory traversal and --sort, settled with
the user: worker pool over directories + one emission-ordering layer owning
the sink; four --sort modes (none/dir/subtree/tree) as a rising-cost
spectrum; single -j/--jobs knob for walk + exec; mode-scoped defaults
(find/fd/rg = all cores + none; modern = max(1,min(cores-1,15)) + dir);
concurrency correctness for prune/quit/depth/exit-code; a separate
--config=tsan + clang-tsan CI cell landed with the threads; phased as a
chain of stacked PRs. No code yet -- this is the spec to build to.
helly25 added a commit that referenced this pull request Jun 27, 2026
* engine: parallel directory traversal + --sort modes + -j (#43)

Add a bounded worker pool (ReadPool) that runs readdir+lstat off the
coordinator thread; the visitor still runs single-threaded in --sort order,
so evaluate/emit/exec/capture/summary stay unchanged and race-free. Workers
are pure (path -> stat'd listing), touching only the thread-safe VFS and a
mutex-guarded queue.

--sort grows to none|dir|subtree|tree (name aliases dir): none = readdir
order; dir = each directory's sorted listing block then its subtrees; subtree
= sorted non-dir entries then contiguous subtrees; tree = total path order.
The ordered modes are deterministic (siblings consumed in sorted order while
the pool prefetches their reads). -j N / --jobs=N sets the worker count
(default 1 = sequential); the parent batches subdirectory reads so the pool
overlaps their IO.

Adds --config=tsan and a ubuntu-only, repo-cache-only clang-tsan CI cell
(kept off the disk cache so the 10 GB Actions budget stays healthy next to
asan). walk_test covers parallel set-equality and deterministic ordering for
all modes at workers=1 and 4; full suite is green under asan+ubsan and tsan.

Deferred (follow-ups): mode-scoped auto-defaults (modern -> parallel+dir) with
the mode mechanism (#54); completion-order subtree streaming; per-worker eval.

* docs: note v1 scope in parallel-traversal design

Record the three deliberate v1 simplifications: IO-only parallelism with a
single-threaded visitor, deterministic (sorted) ordered modes, and opt-in
-j/--sort with mode-scoped auto-defaults deferred to the mode mechanism (#54).
helly25 added a commit that referenced this pull request Jun 27, 2026
…unordered) (#144)

* engine: mode-scoped traversal defaults (modern parallel+sorted, find unordered)

Completes the deferred piece of the parallel walk (#43/#54). RunFind gains an
optional registry::Style: when the user gives no --sort / -j, the default is
mode-scoped - kXff (modern) sorts each directory (--sort=dir) and runs a capped
parallel walk (max(1, min(cores-1, 15))); kFind matches find (unordered) but
saturates cores. std::nullopt keeps the conservative unordered + sequential
default, so the ~32 in-process RunFind callers and conformance are untouched;
the CLI passes the active style (config::ActiveStyle / argv[0] dispatch).

run_test::ModeScopedSortDefault locks the behavior (kXff deterministic dir-
sorted; kFind same set, unordered). Full suite green under default and
asan+ubsan; tsan unaffected (test callers stay sequential; walk_test already
covers the parallel path).

* test+style: use unqualified ElementsAre; document the no-::testing-prefix rule

run_test's ModeScopedSortDefault wrote ::testing::ElementsAre inline even
though the file already has `using ::testing::ElementsAre;` - use the bare
name. STYLE_CPP.md now states the rule explicitly: bring matchers in with a
using and never write the ::testing:: prefix inside an EXPECT_THAT/ASSERT_THAT
expression (fixture utilities like ::testing::Test are exempt). Audited the
suite: this was the only inline-qualified matcher.
helly25 added a commit that referenced this pull request Jun 28, 2026
…oadmap decisions (#193)

Per the directive to put enforcement rules in the repo, not just memory:

AGENTS.md gains two sections:
- Self-documenting features: the registry + globals are the doc SOT; every
  feature add/change updates its Descriptor.summary / GlobalFlag entry +
  kHelpText in the same change, so --help/--man/--markdown stay complete.
- CLI conventions: flag scope by dash count (--global vs -primary; -h/-q/-help/
  -version are special-cased compat globals); flag-only, no subcommands; and a
  user-toggleable boolean capability is a --feature, not a one-off flag.

TODO.md roadmap decisions (design phase, accounting for existing code):
- #43 parallel traversal + --sort: already BUILT (ReadPool, sort modes, tests,
  tsan cell); remaining is a CLI bashtest.
- #45 --exact/--path-encoding: default = filesystem-native (natural per-platform
  case behavior); --exact forces verbatim byte matching; --path-encoding=raw|escape.
- #73 --feature: PARKED (no customer yet) with the full ready-to-build design +
  a trigger (first boolean capability builds it); trigger mirrored in AGENTS.md.
- #54 mode mechanism: subsumed by --config (no --mode flag; --modern deferred).
helly25 added a commit that referenced this pull request Jul 6, 2026
It was a sub-clause buried in the (shipped) #43 parallel-traversal bullet, so
it had no findable standalone location. Give it its own roadmap-tail bullet and
point the #43 record at it, so every tracked item maps to its own line.
helly25 added a commit that referenced this pull request Jul 6, 2026
…280)

* docs(TODO): give #109, smart-case, and #84 their own TODO.md entries

These three were tracked only in the session task list / memory with no
TODO.md home. Add roadmap-tail bullets so TODO.md is complete:
- Hash-verification workflow (#109) [DISCUSS] - -eval matcher + read-into-var
  + --summary tallies; the hashing primitives already shipped (#105).
- Smart-case matching (--smart-case) [DISCUSS] - rg/fd convention; sequence
  before the flavor feature-map help row.
- Sharded-file support (#84) TBD - needs a design pass.

Fold in a correction: the old 'Per-file content hashes' bullet was stale -
the {hash} field + -hash action shipped in #105 (fields.cc, registry.cc);
its only still-open aspect (verify/match) is now the #109 entry.

* docs(TODO): promote the --sort/-j CLI bashtest to its own bullet

It was a sub-clause buried in the (shipped) #43 parallel-traversal bullet, so
it had no findable standalone location. Give it its own roadmap-tail bullet and
point the #43 record at it, so every tracked item maps to its own line.
helly25 added a commit that referenced this pull request Jul 7, 2026
sort_test.sh drives the real binary over a fixed tree: every --sort mode
(none/dir/subtree/tree) walks the whole tree; --sort=tree is a deterministic
global order that is byte-identical across worker counts (-j1 vs -j8, the core
parallel-determinism guarantee); --sort=dir orders each directory's entries; and
--jobs=all / -j N visit everything. The engine unit tests already cover ordering
across worker counts + a tsan cell; this adds the missing CLI-level coverage.

TODO.md: #43/#27 marked complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant