engine: parallel directory traversal + --sort modes + -j (#43) - #140
Merged
Conversation
Add a bounded worker pool (ReadPool) that runs readdir+lstat off the coordinator thread; the visitor still runs single-threaded in --sort order, so evaluate/emit/exec/capture/summary stay unchanged and race-free. Workers are pure (path -> stat'd listing), touching only the thread-safe VFS and a mutex-guarded queue. --sort grows to none|dir|subtree|tree (name aliases dir): none = readdir order; dir = each directory's sorted listing block then its subtrees; subtree = sorted non-dir entries then contiguous subtrees; tree = total path order. The ordered modes are deterministic (siblings consumed in sorted order while the pool prefetches their reads). -j N / --jobs=N sets the worker count (default 1 = sequential); the parent batches subdirectory reads so the pool overlaps their IO. Adds --config=tsan and a ubuntu-only, repo-cache-only clang-tsan CI cell (kept off the disk cache so the 10 GB Actions budget stays healthy next to asan). walk_test covers parallel set-equality and deterministic ordering for all modes at workers=1 and 4; full suite is green under asan+ubsan and tsan. Deferred (follow-ups): mode-scoped auto-defaults (modern -> parallel+dir) with the mode mechanism (#54); completion-order subtree streaming; per-worker eval.
Record the three deliberate v1 simplifications: IO-only parallelism with a single-threaded visitor, deterministic (sorted) ordered modes, and opt-in -j/--sort with mode-scoped auto-defaults deferred to the mode mechanism (#54).
helly25
enabled auto-merge (squash)
June 27, 2026 11:37
helly25
added a commit
that referenced
this pull request
Jun 27, 2026
…g = StatusOr (#141) Follow-up to #140 addressing review feedback: - Listing was a hand-rolled StatusOr (vector + status with an implicit not-ok-means-empty invariant); make it absl::StatusOr<std::vector<Stated>>, matching ReadDir/Stat everywhere else. - ReadPool now uses absl::Mutex with full thread-safety annotations: queue_/stop_ are ABSL_GUARDED_BY(mutex_); the wait predicate Pending() is ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_); Submit/Run are ABSL_LOCKS_EXCLUDED. absl::Mutex::Await replaces the condition_variable (a job becoming ready or stop_ wakes waiters on unlock), so there is no separate signal to forget. Builds clean under -Wthread-safety; full suite green under default, asan+ubsan, and tsan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flagship parallel traversal (#43), built to docs/design-parallel.md. Single PR covering the worker pool + all four
--sortmodes +-j+ the TSan cell, per request.What's in it
ReadPool) runsreaddir+lstatoff the coordinator thread. Jobs are pure (path -> stat'd listing) and touch only the thread-safevfs::FileSystemand a mutex-guarded queue. The visitor stays single-threaded on the coordinator in--sortorder, so evaluate/emit/exec/capture/summary are unchanged and race-free (run.ccuntouched).--sort=none|dir|subtree|tree(namealiasesdir):none= readdir order;dir= each directory's sorted listing block then its subtrees;subtree= sorted non-dir entries then contiguous subtrees;tree= total path order. Ordered modes are deterministic (siblings consumed sorted while the pool prefetches their reads).-j N/--jobs=Nsets the worker count (default1= sequential walk). The parent batches its subdirectories' reads so the pool overlaps their IO; each directory is read exactly once.--config=tsanin.bazelrc+ a ubuntu-only, repo-cache-onlyclang-tsanCI job wired intodone(no multi-GB disk cache, so the 10 GB Actions budget stays healthy beside the asan caches).Tests
walk_test: parallel set-equality (workers=8) and deterministic ordering for all three ordered modes atworkers=1and4.--config=clang --config=asan(asan+ubsan), and--config=clang --config=tsan(no data races).--sort=tree/dir/subtree,--jobs=4,-j2).Deferred (follow-ups, noted in the design doc)
dir; find/fd/rg -> all cores +none) with the mode mechanism (engine: wire -regex captures into gated -exec {0}..{N} #54).