refactor(cli): single anchored walk seam — library globsets are dead and files are double-stat'd after #489

[dekobon]

Summary

#489 moved include/exclude filtering to the CLI walk seam:
expand_seed_paths (big-code-analysis-cli/src/lib.rs) now walks each
directory seed with ignore::WalkBuilder, filters each candidate at the
walk root via walk_seed::match_path_for + WalkFilters::passes, and
passes the resulting flat file list to the library with
empty include/exclude globsets (FilesData { include: GlobSet::empty(), exclude: GlobSet::empty() }). This fixed the
path-form bugs (#488/#489) but left two rough edges worth a
follow-up.

1. Dead library globsets + double per-file stat

The library explore() (src/concurrent_files.rs) still runs
WalkDir::new(path) on every entry it is handed and still applies its
own globset filter (Filters::matches). With a pre-expanded file list
and empty globsets, that filter is a guaranteed no-op and each file is
re-stated a second time (once by the ignore walk in
expand_seed_paths, once by the library WalkDir). Not a full second
tree traversal (single-file WalkDir yields one entry), but O(N)
redundant stats and two filtering mechanisms for one job.

2. Latent re-introduction of the #488 path-form bug

The empty-globset hand-off is load-bearing and undocumented at the
FilesData construction site beyond a comment: the library globset
matcher matches against the walker's emitted (possibly absolute) path
form — exactly the path-form dependence #488 removed. Any future code
path that injects files into FilesData.paths after expand_seed_paths
(or sets non-empty library globsets expecting them to apply) silently
re-inherits the flood-the-offender-set bug. [check.exclude] (#493) is
the existing instance of a filter matching the emitted path instead of
the anchored form.

Suggested direction

Consolidate to a single anchored walk seam: have the library walk accept
an already-resolved file set as terminal (no re-walk, no globset params),
or push the anchored match_path_for filtering into the one library
walk so the emitted path is canonical root-relative once and every
consumer (include/exclude, [check.exclude] #493, baseline keys) matches
that single invariant — removing the per-filter/per-mechanism anchoring
(reanchor_seed for emission + match_path_for for matching + the dead
library globsets).

Acceptance

• One walk + one filtering mechanism; the library globset params are
  either removed or genuinely used.
• No O(N) redundant per-file stat.
• A new path-consuming filter cannot silently re-inherit the #488
  path-form dependence.

Found during the final /code-review of the gate-consolidation epic
(#482). Introduced by the #488/#489 rework; not a correctness bug
today (the CLI pre-filters correctly), but an architecture/perf
follow-up. Related: #493.

---

Status (updated)

Scope: 2.0-deferred (API reshape). Investigation showed the
structural acceptance criteria need a SemVer-breaking FilesData reshape
and/or moving gitignore-aware walking into the library, and that
canonical emission conflicts with the #488 PyO3-parity contract — see the
analysis comment below. The 1.x-safe part (documenting the path-form
invariant at the public API surface, item 2) landed in 997720dc. The
structural single-seam consolidation remains open as a 2.0 item.

[dekobon]

Investigation: this is a 2.0-scoped API reshape, not a 1.x fix

Tracing the architecture confirmed the structural acceptance criteria
("one walk + one filtering mechanism", "no O(N) redundant stat", the
library globsets "removed or genuinely used") cannot be met in 1.x
without a SemVer break or a risky published-behavior change:

1. FilesData is published API and not #[non_exhaustive]
   (src/concurrent_files.rs; re-exported at src/lib.rs:236). Removing
   the include/exclude globset fields, or adding a "terminal file
   set, do not walk" field, breaks external struct-literal construction —
   a 2.0 change per STABILITY.md.
2. Gitignore-aware walking lives in the CLI, not the library.
   expand_seed_paths walks with ignore::WalkBuilder
   (.git_ignore/.git_exclude/…); the library explore() walks with
   plain walkdir::WalkDir (concurrent_files.rs:133), which is NOT
   gitignore-aware. The CLI pre-walk is therefore load-bearing: it
   applies gitignore filtering and hands the library an already-resolved
   file list with empty globsets. Collapsing the double-walk requires
   either moving gitignore-aware walking into the library (changes
   published walk behavior) or a FilesData reshape (item 1) — both
   2.0.
3. Canonical emission conflicts with the fix(cli): manifest paths=["."] resolves absolute and defeats --exclude-from glob matching #488 parity contract. The
   issue's "emit canonical root-relative paths once" would change the
   emitted name/path form, but fix(cli): manifest paths=["."] resolves absolute and defeats --exclude-from glob matching #488's emission fix deliberately keeps
   single-file --paths seeds in their as-given (often absolute) form so
   bca metrics --paths <file> matches the single-file bca.analyze()
   PyO3 API. So emission must stay seed-form; per-consumer match-time
   anchoring (match_path_for, anchor_against_seeds) cannot be fully
   unified away while that contract holds.
4. The O(N) "double stat" is inherent to the two-walker split — the
   library's WalkDir::new(file) must stat the entry to yield it, and
   determining file-ness to skip it is itself a stat. Not cheaply
   removable without item 1/2.

Landed in 1.x (commit 997720dc)

The one safe, behavior-preserving part — item 2 of the issue
("undocumented load-bearing invariant") — is fixed: FilesData's public
docs now state that include/exclude match the emitted (root-
prefixed, path-form-sensitive) path, that a ./-anchored glob will not
match an absolute walk root, and that the CLI pre-filters + passes empty
globsets precisely to avoid this. A future/external caller now has the
invariant at the API surface (docs.rs), so they cannot silently
re-inherit the #488/#489 path-form dependence unaware.

Deferred to 2.0 (structural)

Items 1 and the "single anchored walk seam / one filtering mechanism /
no double-walk" goal require the FilesData reshape (terminal,
pre-resolved, anchored file set — no library re-walk, globsets removed)
plus moving gitignore-aware walking to the producer. That is a
deliberate 2.0 API change and should be planned alongside the other
deferred 2.0 shape changes, not slipped into a 1.x minor.

Leaving open as a tracked 2.0 architecture item. The 1.x
defensive doc has landed; the structural consolidation is the remaining
work.

[dekobon]

Resolved — implemented the 2.0 single-seam reshape (commit c57058d4, review fix 2b293816)

With breaking changes authorized for a 2.0, the structural consolidation is done.

Library (src/concurrent_files.rs) — now a pure concurrent file-set processor:

• FilesData drops the include/exclude GlobSet fields → FilesData { paths: Vec<PathBuf> }, a resolved terminal file list.
• Removed the walkdir re-walk (walk_dir_files), Filters/is_hidden, the proc_dir_paths/proc_path dual callback, and the all_files HashMap return. ConcurrentRunner::run → Result<(), ConcurrentErrors>.
• walkdir + globset moved to dev-dependencies (no production use left).

CLI — single anchored walk seam: expand_seed_paths (gitignore-aware, walk-root-anchored, #488/#489) is now the only walk; the library just processes its output. The preproc basename grouping moved into the CLI (group_files_by_basename, computed from the resolved file list).

Acceptance — met

• ✅ One walk + one filtering mechanism (the CLI walk; the library no longer walks or filters).
• ✅ No O(N) redundant per-file stat (library no longer re-walks).
• ✅ Library globset params removed (not dead, not path-form-sensitive).
• ✅ A new path-consuming filter cannot silently re-inherit the fix(cli): manifest paths=["."] resolves absolute and defeats --exclude-from glob matching #488 dependence — there are no library globsets to misuse, and the CLI's anchored walk is the sole filter.

Bonus: fixed a latent #489 regression

Tracing the dual-callback revealed that since #489 the CLI passed a flat file list, so proc_dir_paths/process_dir_path never fired and run_command_preproc handed fix_includes an empty grouping — silently breaking bca preproc cross-file #include resolution (the smoke test missed it: no #include fixtures). Moving the grouping to the CLI (from the file list it already has) fixes it. Guarded by a new preproc_resolves_cross_file_include_across_directory test (test-via-revert verified).

Constraints honoured

Emission is unchanged (no .bca-baseline.toml key shifts, no insta/submodule snapshot drift) — only the library's redundant re-walk was removed. Web + PyO3 are unaffected (neither uses ConcurrentRunner). Recorded (breaking) / 2.0-deferred in CHANGELOG.md + STABILITY.md. make pre-commit green (incl. cargo +nightly udeps, both self-scan tiers, make py-test).

The 1.x interim doc note (commit 997720dc) is superseded — the globset fields it documented are gone.