feat(scan): LLM-scale corpus scanning — Tiers A/B/C (stats, digest, search-text, summarize, topics, clusters)#82
Merged
Merged
Conversation
…iers A+B)
Add three read-only, LLM-facing capabilities for models with tens of
thousands of requirements, each a first-class CLI subcommand mirrored by a
read-only MCP tool (and on the run_report allow-list):
- stats (REQ-TRS-OUT-021): corpus-shape facet histograms (status, reqDomain,
sil/asil, package, tags) plus coverage and orphan rollups; --group-by,
--where/--status/--tag scoping, --config projection lens. Reuses
coverage_summary and the validator reverse indices; coverage numbers equal
matrix. ADR-SYS-STATS-001.
- digest (REQ-TRS-OUT-022): token-budgeted, one compact line per requirement
(NDJSON, or {total,offset,rows} under --json), cursor-paged, sharing the
stats scoping. The "dump the slice" companion to stats.
- search-text (REQ-TRS-SEARCH-001): ranked full-text over element body+name via
an in-memory Okapi BM25 inverted index (no external engine, no persisted
index — ADR-SYS-SCAN-001), best-first with marked snippets. Complements the
identifier-fuzzy search tool.
Requirement-first: REQ-TRS-OUT-021/022, REQ-TRS-SEARCH-001 (+ SEARCH prefix)
with TC-TRS-OUT-021/022 and TC-TRS-SEARCH-001. New integration harnesses
(tests/{stats,digest,ftsearch}.rs, 29 tests). qual/ validates clean; full
suite, clippy and workspace build green.
Add three deterministic, offline, read-only capabilities that convey what a
requirement corpus *means* (Tier C), each a first-class CLI subcommand mirrored
by a read-only MCP tool. A shared textstats module owns the TF-IDF core (with a
stopword filter so labels are content words, not "the"/"shall").
- summarize (REQ-TRS-OUT-023): bottom-up per-package rollup — count, status
split, TF-IDF "about" terms, one-line extracts of representative requirements
— nested through the hierarchy. Extractive (not an LLM summary); each node's
fields are cached under .syscribe/cache/summaries.json keyed by a subtree
content hash, with a cache-version guard so a logic change invalidates a stale
hit. --scope/--depth/--config. On the run_report allow-list.
- topics (REQ-TRS-SEARCH-002): distinctive per-package keywords via TF-IDF,
demoting vocabulary common to every package.
- clusters (REQ-TRS-SEARCH-003): topical clustering via TF-IDF cosine k-means,
surfacing cross-package themes. Deterministic farthest-first init (no random
seed); --k validated (>=1) and clamped to the element count.
Design ADR-SYS-SCAN-002 records the deliberate offline/deterministic choices:
extractive (not LLM) summaries and TF-IDF cosine (not neural embeddings), so the
tool stays a single offline binary; both heavier options are deferred as opt-in.
Requirement-first: REQ-TRS-OUT-023, REQ-TRS-SEARCH-002/003 with TC-TRS-OUT-023,
TC-TRS-SEARCH-002/003. New harnesses (tests/{summarize,topics,clusters}.rs, 15
tests). qual/ validates clean; full suite (147), clippy and workspace green.
Promote the six requirements added in this PR out of draft to `verified`
(each has a passing verifying TC-TRS-*), and their six verifying TestCases to
`approved`, so coverage counts them as verifying rather than planned.
`verified` (not `approved`/`implemented`) is deliberate: these are tool
self-requirements qualified by TestCase verification, not by an architecture
`satisfies:` layer — the whole qual/ self-model has no architecture model — so
`approved`/`implemented` would trip W300 ("no satisfying architecture element")
on every one. `verified` reflects reality and keeps qual/ clean: 0 errors, no
W300/W305 on the promoted set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a progressive, read-only, offline, deterministic toolkit for letting an LLM scan and summarize a model with tens of thousands of requirements without ever reading 15k files. The guiding pattern is aggregate → narrow → drill → summarize. Every capability is a first-class CLI subcommand and a read-only MCP tool returning the same JSON as
--json; the reporting ones are on therun_reportallow-list.Built requirement-first (7 new
REQ-TRS-*, 7 newTC-TRS-*, 3 ADRs) with new integration harnesses.qual/validates clean; the fullsyscribesuite (147 tests), clippy, and the workspace build are green.Tier A — shape (
ADR-SYS-STATS-001)stats(REQ-TRS-OUT-021) — corpus-shape facet histograms (status, reqDomain, sil/asil, package, tags) plus coverage & orphan rollups;--group-by,--where/--status/--tag,--configlens. Reusescoverage_summary+ validator reverse indices, so coverage equalsmatrix.Tier B — slice & retrieve (
ADR-SYS-SCAN-001)digest(REQ-TRS-OUT-022) — token-budgeted, one compact line per requirement (NDJSON or{total,offset,rows}), cursor-paged, sharing thestatsscoping.search-text(REQ-TRS-SEARCH-001) — ranked full-text over body+name via an in-memory Okapi BM25 inverted index (no external engine, no persisted index), best-first with marked snippets.Tier C — meaning (
ADR-SYS-SCAN-002)summarize(REQ-TRS-OUT-023) — bottom-up per-package hierarchical rollup (count, status split, TF-IDF "about" terms, representative one-liners), extractive (not an LLM summary), with a content-hash cache (version-guarded) under the git-ignored.syscribe/cache/.topics(REQ-TRS-SEARCH-002) — distinctive per-package keywords via TF-IDF.clusters(REQ-TRS-SEARCH-003) — cross-package topical clustering via TF-IDF cosine k-means, deterministic (farthest-first init, no random seed).Design decisions worth a look
Two deliberate deviations keep the tool a single offline binary (documented in the ADRs; both heavier options deferred as opt-in):
summarizeemits is exactly the scaffold an LLM abstractively summarizes from.Notes for reviewers
textstatsmodule owns the TF-IDF core; a stopword filter keeps labels as content words (not "the"/"shall") whilesearch-textkeeps its literal tokenizer.🤖 Generated with Claude Code