perf: counter fast path, single-fd reads, parallel directory walk#2
Merged
Conversation
classify() ran prefix checks against every delimiter list at every non-space byte, costing 4-7 memequal calls per ordinary identifier byte (21% of CPU samples on GOROOT/src). A precomputed [256]bool gate marks the bytes that can start any delimiter; everything else is consumed by a one-load-per-byte skip loop before falling through to the unchanged check sequence. Also rewrites the inner searches around bytes.Index/IndexByte: block comment closers in non-nested languages jump straight to the closer, and indexDelim anchors on the delimiter's first byte with a backslash-parity check (delimiters starting with a backslash keep the old scan). Semantics are unchanged; a verbatim copy of the previous implementation ships as a differential oracle exercised over GOROOT/src, curated edge cases, and seeded random delimiter soup for every builtin language. Benchmark (Apple Silicon, GOROOT/src end to end): user CPU -70%, counter.classify 21.4% -> 4.4% of samples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reading a file cost up to eight syscalls: os.ReadFile's open/fstat/read/read/close, plus a whole open/read/close shebang sniff for unknown extensions. countFile now opens each file once, sniffs the shebang from the first read of the same descriptor when needed, and reads the content into a per-worker scratch buffer (grown on demand, shrunk after outsized files). EOF is confirmed by an explicit 0-byte read so short reads on FUSE/network filesystems stay correct. Raw descriptors are unix-only; other platforms keep os.File. The walk itself was the pipeline's critical path: one goroutine's ReadDir latency inflates ~4.5x under concurrent worker I/O, leaving the jobs channel empty in up to half of occupancy samples. Directory recursion now hands subtrees to a bounded pool (min(8, jobs)) when walk order is unobservable: parallel traversal preserves the final report byte for byte (Build sorts everything; dedup buffers then sorts), so it is enabled only when Warn and Trace are unset and symlinks are not followed, and each root drains before the next starts so overlapping roots keep their serial outcome. Measured on Linux 6.9 (84k files): wall 8.8s -> 3.8s, user CPU 25.8s -> 6.0s, heap allocations -63%, GC cycles 64-68 -> 23-26. Output verified byte-identical across aggregate/by-file/dedup/yaml, overlapping roots, -vv trace, and unreadable-file warnings; 10-run determinism check stable; tests pass under -race. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three high-severity findings from the performance audit, productionized. Output is byte-identical to main in every mode; the speedup on the Linux 6.9 tree (84k files, 1.5 GB) is 2.3×, which moves aloc from slowest to fastest of the modern counters:
What changed
1.
counter: per-language first-byte gate table (commit 1)classify()prefix-checked every delimiter list at every non-space byte — 21% of CPU samples on real code. A precomputed[256]boolgate lets ordinary identifier bytes skip all delimiter checks at one array load each; block-closer and string-delimiter searches now usebytes.Index/IndexBytewith a backslash-parity escape check. −70% user CPU on GOROOT/src.2.
walker: single-descriptor pooled reads (commit 2)os.ReadFile+ the separate shebang-sniff open cost up to 8 syscalls per file; now each file is opened once and read into a per-worker reusable buffer: −63% heap allocations, GC cycles 64–68 → 23–26. EOF is confirmed with an explicit 0-byte read so FUSE/NFS short reads stay correct. Raw descriptors are gated//go:build unix; other platforms keepos.File(cross-compiles verified for windows/linux).3.
walker: bounded parallel traversal (commit 2)The serial walk starved workers (jobs channel empty in up to 52% of samples). Directory recursion hands subtrees to a bounded pool of
min(8, jobs)goroutines — only when walk order is unobservable:Warn/Traceunset, symlinks not followed, and each root drains before the next so overlapping roots keep their serial outcome.-v/-vvruns keep today's fully deterministic walk.Correctness evidence
-raceon walker/cli/counter: green.old_impl_test.goand compared against the new one over all of GOROOT/src (~7k files), 26 curated edge cases, and 200 seeded-random delimiter-soup inputs per builtin language.--by-file/--dedupJSON and YAML over three corpora (incl. Linux kernel), overlapping relative+absolute roots,-vvfull trace, and unreadable-file warning text.Full audit (17 verified findings; follow-up PRs coming for the rest): see the session artifact.
🤖 Generated with Claude Code