Repository navigation
SigMap 8.81.0
Minor. The retrieval index keeps up to 200 signatures a file instead of 25, so verify and search_signatures find the symbols ranking never needed; a second run costs a sixth of the first; and extraction recall is measured against CPython's ast and Ruby's Ripper, with floors in CI (#982, #971, #983). Retrieval does not move by construction: what the ranker reads of each file is byte-identical to v8.80.1's.
Added
- An extraction cache,
.context/extract-cache.json(#971, PR #1016) — on by default,extractCache: falseturns it off. Per file it keeps what a run derives from the content (the signatures, imports, line and token counts, the module-doc line, TODO lines), keyed by the file's SHA-1 and by a fingerprint of everything else the answer depends on: SigMap's version, the extractor sources, the generator itself, the extraction ceiling,depMap,secretScan, and the Python interpreter and what selects it (PYENV_VERSION,VIRTUAL_ENV,CONDA_PREFIX,.python-version). A file whose size and modification time are unchanged, and older than the entry by git's "racily clean" margin of two seconds, is not read at all. The git answers behind## recent changesand the hot-file ordering are kept per commit range (the ordering also by the shallow boundary). Credentials are redacted before anything is kept. Not used withexactness.typescript|lsp|scip, whose answers come from outside the file. On Django, a re-run with nothing changed costs 15–17% of the first run's CPU time withpython3on PATH (0.48 s against 2.90 s, 0.77 s against 5.16 s on two CI runners) and 27–30% without it (0.50 s against 1.68 s, 0.77 s against 2.83 s); one changed file adds 0.01–0.07 s. maxIndexSigsPerFile, default 200, andmorein the index file (#982, PR #1014) —maxSigsPerFile(25) is a token budget for the generated context file, yet the retrieval index inherited it, so on Django the index held 57–76% of the public symbols andverifyflagged real ones. The index file now holds two lists per file:files(the firstmaxSigsPerFile, byte-identical to v8.80.1's on flask, sidekiq, django with and withoutpython3and rails, test and pipeline entries included, so ranking,ask,read_contextandlist_modulescannot change) andmore(the rest, up to the index limit), read only by the tools that look a symbol up:verify,verify-plan,search_signatures,explain_file,get_callee_signatures, the call graph's "not tracked" check and stack-trace squeezing (buildSigIndex(cwd, { full: true })). The watcher's patch,freshenandnotify_file_createdwritemoretoo.npm run validate:extraction-recall(#983, PR #1013) —scripts/run-extraction-recall.mjsasks CPython'sastand Ruby's Ripper what each file of the pinned Flask, Django, Rails and Sidekiq clones defines, and counts how many of those names SigMap lists, per kind, for the extractor's output and for the index as shipped. Floors live inbenchmarks/extraction-recall-floors.json, one entry per reference version prefix (a run is compared only with the floors recorded for its own version);--require-reference, which CI passes, fails unless every language with floors was measured on a covered version. The 36 other languages are listed as unmeasured. Dev-only: nothing ships in the npm package. Guide:docs-vp/guide/extraction-recall.md.npm run benchmark:perfandnpm run validate:warm-cache(#971, PR #1016) —scripts/run-perf-benchmark.mjsmeasures a re-run as CPU time (user plus system) of the generator process tree, the median of several runs on a fresh copy, with the machine and the load average beside every figure and the rivals' README claims labelled as claims, not runs.--equivalenceruns, on every clone underbenchmarks/repos, the same sequence with the cache on and off (first run, second run, after one changed file) and fails unless the context file and the retrieval index are identical; it is the last step ofvalidate:benchmark-determinismand a step of thexrepoCI job, which also uploads the cost report (report-only). 50 of 50 local clones and 25 of 25 in CI.- Tests —
warm-cache.test.js(39 cases; a sweep of 47 one-line mutants of the cache, the git memo, the directory memo, the Python identity and the run summary is caught in full),index-limit.test.js(38; all fail on the old code),extraction-recall.test.js(30; a sweep of 36 mutants of the harness, the floors logic and the CI step is caught in full).
Fixed
- The Python AST tier stopped at 30 signatures a file without saying so (#982, PR #1014) — and listed a class's constants ahead of its methods. It stops at the regex tier's 200 and ends with the same
… +N more signaturesnotice. Found by the #983 harness. - The index missed most of a large file's symbols (#982) — Python with
python3on PATH, recall againstastof the index as shipped, class / function / method: 64.0 / 87.9 / 57.5 % to 98.8 / 100.0 / 98.6 %; withoutpython3: 65.0 / 77.3 / 57.1 % to 99.3 / 98.5 / 90.6 % (the regex extractor's own limit: it lists none ofModelAdmin's members incontrib/admin/options.py); Ruby over the 298 files its index holds, class / module / method / method under its owner: 92.1 / 98.7 / 88.0 / 88.0 % to 100 / 100 / 98.3 / 100 %. What is left withpython3is the extractors' own 200: two of the 266 files hold more than 200 symbols. - The run summary graded a complete index "D" (#971) — on Django
Coverage : D (5%) — 139 of 3014 source files included, though the retrieval index holds every file with symbols. It now readsIndexed : 2,069 of 2,069 files with symbols (100%)andIn context : 139 of 3,014 source files (5%) fit the context file, neither graded by the other. The strategies that have no single budgeted file (index,per-module,hot-cold) say so instead of claiming one (the old line was inflated for them too), andSymbols foundcounts every file, not the part that fit the budget (1,488 on Django). Pinned bymonorepo.test.jsandwarm-cache.test.js. - The grounding-coverage benchmark counted a design the tool no longer has (#982, PR #1017) —
npm run validate:grounding-coverageread the firstmaxSigsPerFilesignatures of each file whileget_callee_signatureslooks a symbol up infilesandmore, and the universe is whatever the extractors list (up to 200 a file). Once the Python AST tier listed past 30, a file past 25 symbols read as ungrounded and click (74.9% against a floor of 89.0%), httpx (79.4% < 90.0%) and rich (73.3% < 88.0%) failed the gate on merged develop. The gate runs in no CI job, so the release prep found it. It now countsbuildSigIndex(repo, { full: true }): 43 repositories pass, the aggregate reads 100.0% (35,358 of 35,362) against 74.4% (26,299 of 35,362), and the floors are re-recorded with the gate's own--save-floors(5 points below measured, capped at 0.95). A 40-function fixture reads 25 of 40 on the old code. - Test files were extracted one
python3process at a time (#971) — 176 processes on Django withpython3on PATH, because only the main loop prefetched; the files that are not cached are extracted in one batch. KNOWN_LIMITATIONS.mddescribed truncation wrongly (#982) — it promised a… +N morenotice at 25 signatures and "8 members per class"; it now states the three limits, and a test pins every number to the code.
Changed
- The index file is 13–57% larger (#982) — Django 1.01 to 1.57 MB (1.04 to 1.64 MB with
python3), Rails 0.49 to 0.68 MB, Flask +21%, Sidekiq +13%.askis unchanged within noise (Django best of five, 598 to 602 ms). The generated context file is unchanged: its sha256 is equal on flask, django (both Python tiers), sidekiq and rails.askstill prints at most 20 signatures a file;explain_fileandsearch_signaturescan print up to 200 for a very large file. - Fixed costs a warm run still paid (#971) — one directory listing per run instead of up to three (9,116 listings of 3,532 directories on Django), one framework-detector walk instead of one per name (160–340 ms), module resolution memoised (13,000 filesystem probes a run), and a depth-1 clone no longer probes nine ancestors with nine
gitprocesses for a section it then leaves empty. - The published grep scan does not count the extraction cache (#971, PR #1018) — the honest benchmark's scan counts SigMap's own files on purpose, so its figure stays comparable between releases (the autopsy records what that costs grep). The cache is a new file of that kind in every indexed clone, and counting it moved the grep baseline from 39.2% to 34.4% and the headline lift from 2.37× to 2.70× with SigMap's 92.8% unchanged: SigMap's files took 187 of the 625 top-5 places instead of 129.
.context/extract-cache.jsonis left out of the ranking and the file count; everything else is counted as before. A control tree of merged develop reads 40.0% (50 of 125), lift 2.32×, 132 places; the scan with SigMap's files left out reads 46.4% (2.00×) either way. - Default MiniMax live ablation runs to MiniMax-M3.1-Flash (PR #1015) — thanks @octo-patch. Flash is listed before M3 in the pricing choices, explicit legacy/custom models and project overrides are preserved, and offline request coverage is added.
Notes
- Upgrading. Nothing to configure. The first run after upgrading is a cold one and writes
.context/extract-cache.json(inside the self-ignoring.context/). A repository that lowered onlymaxSigsPerFileto keep the index small now gets a 200-signature index; setmaxIndexSigsPerFiletoo (it never goes belowmaxSigsPerFile). The summary'sCoverageline is replaced byIndexedandIn context. - Criteria not met. #982 asked for at least 95% on functions, classes and methods: met on the default path and for Ruby, not for Python methods without
python3(90.6%, the regex extractor). #971 asked for a warm run at no more than 25% of cold: met withpython3(15–17%), not without it (27–30%): the first run is cheap there, so the fixed floor of about 0.5 s (Node start-up and compiling the 2.2 MB bundle, one listing of the tree, config detection, fourgitprocesses) is a larger share. Both issues stay open for the maintainer's decision. - Known limits of the cache, documented in
docs-vp/guide/config.md#extractcache: a file whose content changes while its size and modification time do not (a tool that restores timestamps) is not noticed until one of them moves; a file that becomes unreadable is still served until it changes; the git memo does not see git's own configuration, a nested.gitattributes,.git/info/attributesorgit replace. Deleting.context/extract-cache.jsonis always the way out. - Found, not fixed. On
gin(source roots["."]) a run indexes the context file the previous run wrote (118 to 119 files), with the cache off too; test and CI-workflow entries in the retrieval index are never secret-scanned (pre-existing); the Rails index holds 298 of 757 files (#997). - Published metrics. Token reduction (95.7%), retrieval hit@5 (80.4%), task success (61.9%), prompt reduction (44.4%), test-discovery F1 (98.0%), the grounding corpus and the third-party gate reproduce v8.80.1's reports leaf for leaf apart from version and date stamps, step timings and captured output tails (the six reports the matrix rewrites differ in two leaves: the output tail and the worktree path in it). Two readings of this repository's own corpus move, both with the merged code (an untouched merged
developin the same layout reads the same): the honest comparison's single-shot grep baseline, 39.2% (49 of 125) to 40.0% (50 of 125), lift 2.37× to 2.32×, with SigMap's 92.8% unchanged and its MRR 0.758 to 0.767; andhard, 71.1% (64 of 90) on a clean tree of v8.80.1 to 70.0% (63 of 90):h062goes from rank 4 to 7 because the release addedsrc/util/dir-listing.jsand rewrote the header ofsrc/cache/entry.js, which now hold words of its question. The ranker is unchanged and the gate's floor, 70.0%, now has no task of headroom. The like-for-like grep scan (SigMap's own files left out) reads 46.4%, 2.00× (45.6%, 2.04× before).