Skip to content

SigMap 8.83.0

Choose a tag to compare

@github-actions github-actions released this 26 Oct 23:13
· 26 commits to main since this release
45b413c

Minor. Ruby and C/C++ move to the anchored tier, the import graph reads seven more language families, and the call graph and the import graph are measured against what each language's own tooling resolves, with the misses published (#975, #980, #976). This is the first release since v8.81.0 that changes shipped code: src/extractors/ruby.js, src/extractors/cpp.js, src/graph/ and the bundle; the npm package gains two files (222 → 224). Retrieval does not move through the import graph, because the ranker is still handed only the original eight families. The third-party headline moves by one task, 116 → 115 of 208 (55.3%, 95% interval 48.5–61.9%), because the C++ extractor stopped inventing signatures (see Changed).

Added

  • Ruby and C/C++ carry :start-end line anchors and are read as structure (#975, PR #1031) — both leave Tier 3 for Tier 2 ("C# (15)" in the tier table), so get_lines can read one symbol instead of the file. Ruby: every class, module and def is anchored to the line of its keyword and the line of the end that closes it; a reopened class is one entry. Measured against Ripper (Ruby 2.6.10, 757 files): 9,701 of 9,701 anchors, every one equal to the parser's lines; on Ruby 3.2.11 (833 files, which also parses endless def and pattern matching) 12,471 of 12,471. C/C++: the file is tokenised with comments, literals and preprocessor lines masked, brackets are paired, and each scope is read for what it declares: namespaces, extern "C", class / struct / union with members and bases, enum / enum class, functions with a body or only a prototype, constructors, destructors, operators, out-of-line A::b (listed under A) and templates (the anchor starts on the template<…> line); both branches of an #if are listed. Measured against clangd 21.0.0 on 248 files of libuv, spdlog and abseil-cpp, before → after: classes 49.5% → 95.6%, enums 0 of 29 → 93.1%, functions 57.9% → 88.2%, methods 16.6% → 96.6%, methods under the right class 13.0% → 96.9%, and 4,192 of 4,192 emitted definitions anchored, 3,961 of 4,073 equal to clangd's lines. The 200-signature cap is most of what is missing: raised, the same files score 99.2 / 100 / 97.3 / 99.9 / 99.9. Hard fixtures with literal expected signatures (cpp-extractor.test.js, 19 cases; 50 mutants of the reader killed) and a cross-check against clangd where it is installed.
  • A held-out check, and what it found (#975) — four repositories nobody developed against (leveldb, hiredis, yaml-cpp, cJSON; 222 files; fixed commits; node scripts/extraction-recall/cpp.mjs --held-out, clone recipe on the guide page). The first run read functions at 83.7% and found three patterns the development set had not shown: a macro that wraps the return type (CJSON_PUBLIC(T) f()), a macro that stands for it (EMITTER_MANIP f()), and a field a macro annotates (int n_ GUARDED_BY(mu_);, listed as a method; 73 such invented lines went away across the development and held-out files and nothing else did). All three are fixed with a fixture each, so those four repositories are no longer held out for those patterns (93.2 / 97.6 / 98.8 / 99.2 / 98.1 after). Both rows are committed in benchmarks/reports/extraction-recall-cpp-held-out.json. C/C++ has no recall floors (the reference is a language server that CI does not install; without clangd the language is "unmeasured" and the run exits 0). Not read, and stated in KNOWN_LIMITATIONS.md: type aliases (including the anonymous typedef struct { … } name_t;), a return type wrapped by a lower-case macro (PyAPI_FUNC(int) f()), an #if that opens a brace in one branch only (it can swallow the rest of the file), and a C++14 digit separator. No doc hints for either language.
  • The import graph reads C/C++/Objective-C, PHP, C#, Dart, Vue/Svelte/Astro, Lua and GDScript, and answers at module level for Swift (#980, PR #1030) — --impact, get_impact, the blast radius in plan and doctor go from eight import families to fifteen (src/graph/import-resolvers.js; src/graph/swift-modules.js). Precision first: an import that cannot be resolved to exactly one repository file (an external package, a system header, an ambiguous name, a path built at run time) yields no edge. Swift names modules, not files, so --impact on a Swift file names its Package.swift target and lists the files of other targets that import it, labelled module level. Component scripts read import lists that span lines. The ranker is still handed only the original eight families (#934/#935). Measured on nine pinned repositories against clang -E -H, TypeScript's own module resolution, Composer's autoloader and swiftc + SwiftPM (node scripts/run-import-accuracy.mjs, protocol benchmarks/IMPORT_RESOLVER_PROTOCOL.md committed before the scorer ran): C/C++ 99.7% precision / 99.7% recall, PHP 100.0 / 100.0, Swift module level 100.0 / 99.3, Vue/Svelte/Astro 100.0 / 79.1%. Eight predictions were registered; seven held and P4 (component recall ≥ 90%) was missed. Every remaining component miss is in the resolver that all .js/.ts files share (390 of 406 are the svelte package importing "svelte" through its own exports), which was deliberately not changed here because the ranker's graph is built from it. C#, Dart, Lua and GDScript have fixtures and no accuracy figure (no reference could be run). Page: docs-vp/guide/import-accuracy.md; report: benchmarks/reports/import-accuracy.json. Handing the ranker the extended graph lost two third-party tasks in an experiment and was not shipped.
  • The call graph is measured against compiler-derived edges (#976, PR #1029) — node scripts/run-callgraph-accuracy.mjs scores --callers, --callees and get_method_impact against caller-to-callee edges from a SCIP index of the same code, on 10 of 11 registered repositories (gson does not build under scip-java and is reported unmeasured with its error). Plain f() calls are found at Go 99.0%, JavaScript 99.3%, Python 55.7%, TypeScript 26.5%; calls through a receiver are not in the graph outside Java (TypeScript 0.0%, Python 0.1%, Go 0.0%, Java 43.8% on one repository); classes are not nodes. The registered 70% receiver-method-recall threshold tripped in TypeScript, Python, Go and Java, which says a receiver-type pass is warranted and is not one. The protocol (repositories, indexer versions, edge definitions, the threshold, five predictions) was committed before the scorer existed, and the deviation log keeps the first run. No src/ change. Page: docs-vp/guide/callgraph-accuracy.md, quoted where KNOWN_LIMITATIONS.md describes blast radius.
  • run-signal-verdicts.mjs --only <corpus> --save re-scores one signal-arms corpus and keeps the others as recorded (#975) — recorded in benchmarks/reports/signal-arms.json says which release and moment each corpus was scored at. A control on untouched develop already reads the self-scored corpora differently from a saved report, so a change that moves only the third-party corpus must not rewrite all six. The merge is pure and tested (9 mutants killed).
  • scripts/run-extraction-recall.mjs measures C/C++ against clangd (#975) — the third reference after CPython's ast and Ruby's Ripper; two rows are judged on what the extractor emits (anchored, and anchor equal to the parser's lines). Ruby floors rise to 99 for anchors and for the Ruby 3 owner row.

Changed

  • The third-party baseline is re-recorded, 116 → 115 of 208 (55.8% → 55.3%, 48.5–61.9%) (#975) — one spdlog task (x102) was lost, and a control on untouched develop reproduces the old baseline exactly. The cause is the old C++ extractor: for pattern_formatter-inl.h it listed 102 "signatures" such as if(!padding.truncate_), which happened to hold the query's words; the new one lists the 150 declarations the file has, and the task now ranks 7th. Nothing was tuned to win it back. The README, the compare page, benchmarks/latest.json, the retrieval guide and the signal-arm report's third-party column follow (plain 115, mined expansions net 0, body words −37 with 3 won and 40 lost); the other signal-arm columns stand as recorded.
  • Three published figures move, all caused by the two extractors — each leaf of the regenerated reports was diffed against v8.82.0, and a control run of untouched merged develop in the same layout reproduces every one, while a control at the v8.82.0 tag reproduces the saved v8.82.0 reports leaf for leaf. In the 18-repository retrieval corpus abseil-cpp reads MRR 1.000 → 0.733 (two tasks go from rank 1 to rank 3, both still hits; rails and vapor change their signature counts and not a rank), so hit@5 stays 80.4% while the task-success proxy goes 61.9% → 60.0% (65 → 63 of 105 correct) and prompt reduction 44.4% → 43.7% (1.58 → 1.6 prompts a task). The honest comparison reads 92.8% → 92.0% (116 → 115 of 125; the grep baseline stays 40.0%, so the lift is 2.32× → 2.30×): the whole move is the task set scored against this repository itself (19 → 18 of 20), which drifts with the tree by design. Token reduction (95.7%), test-discovery F1 (98.0%), the grounding corpus and the hallucination guard reproduce v8.82.0 leaf for leaf.
  • The quality page quoted grounded-symbol totals that were already stale against the saved report (16,718 and 51,759, with rails at 2,010); they now read the report: 16,259 grounded, 52,218 dark, 5,339 hidden files, rails 1,901 (25%), abseil-cpp 1,402 (12%), vapor 400.
  • The KNOWN_LIMITATIONS.md tier table, the README tier table and the language table move Ruby and C/C++ to the anchored tier; the recall page gains a C and C++ section; C/C++ no longer lists typedef, which was never read.

Fixed

  • Three Ruby misreads that put an anchor on the wrong end (#975) — a symbol ending in = (alias_method :a, :[]= unless x) was taken for an assignment so the unless opened a block; the operator symbol :/ started a regexp that swallowed code up to the next slash; and an if after <<, ,, &&, ||, =>, and/or/not was taken for a modifier. Found by checking anchors against Ripper. Owner recall on Ruby 3 went 99.3% → 100.0% as a side effect, and def str.inspect is listed under the receiver it names.
  • A Dart package's pubspec.yaml is found by the directory's real spelling (#980) — the resolver probed the filesystem with lowercased graph keys, which a case-insensitive macOS hides and Linux does not.
  • Astro frontmatter after blank lines is read, and a recursive component is not its own importer (#980).
  • The accuracy harness's Swift reference writes to a file, not /dev/stdout (#980) — swiftc -o /dev/stdout fails under Node's spawnSync on Linux.

Notes

  • A JS/TS import matcher gap, found and not fixed (#980) — the matcher is line-based, so import {\n a,\n b\n} from './x' gives no edge in a plain .ts/.js file (components are read with a flag). Prettier-wrapped imports are common, so --impact on JS/TS files under-reports importers. It changes what the ranker's graph holds, so it needs its own issue and gate review.
  • The retrieval gate has no headroom: hard is exactly 63 of 90 (70.0%) and two mined tasks swap places by a few thousandths on any change to the indexed file set (m003 ↔ m053), net zero.
  • abseil-cpp is the one retrieval repository that reads worse (MRR 1.000 → 0.733, hit@5 unchanged): its index holds 114 signature lines where it held 130, and for two tasks the file that ranked first (time_zone.h, span.h) now ranks third behind cctz/src/time_zone_format.cc and civil_time_benchmark.cc. The cause was not isolated further, and nothing was tuned.