Tri-comparison chart: drop recall/args ratios for unranked found-counts - #1868
Merged
Conversation
_walk_tree_sitter listed every matching tree-sitter node with no consolidation, so a Haskell function written as multiple pattern-match clauses (idiomatic style -- toJSON instance methods, guard-clause helpers, etc.) counted as multiple distinct occurrences of one name. Confirmed via the ledger's own toJSON example: 5 clause-nodes at consecutive lines in one Options.hs instance block, where GitGalaxy and ctags each correctly report 1. This is the same bug tree_sitter_accuracy_audit.py's measure() already fixed once (#1614-1616) -- this module wrote its own simpler walk instead of reusing that closure and silently reintroduced it. Ported the same fix: consecutive func-type sibling nodes sharing a name (interleaved `comment` nodes don't break the run) collapse into the first occurrence only. Verified: haskell's raw tree-sitter function count drops from 275 to 146 (GitGalaxy's own count is 148), and the ledger's tree_sitter-only-vs-[ctags,gitgalaxy] shape (91 occurrences) no longer reproduces. Regenerated the full ledger and chart across all 45 languages with tree-sitter-language-pack + a locally-built universal-ctags -- no other language's entries were affected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolved without needing a fresh investigation -- two from source already read earlier in this same effort, one from a doc comment already sitting in ctags_reader.py: - agree[gitgalaxy]_vs[ctags,tree_sitter] (2): mixed verdict, read individually. getExtensions is a real instance method both other tools structurally can't see; extensionEnabled is a genuine GitGalaxy false positive (misreads a guard-clause call as a definition). - args agree[none]_vs[gitgalaxy,tree_sitter] (9): one systematic cause, point-free/eta-reduced equations -- GitGalaxy counts true signature arity, tree-sitter counts only the aligned clause's explicit patterns. Same shape as Claim 1, not an engine defect. - class agree[gitgalaxy,tree_sitter]_vs[ctags] (16): not a real discrepancy at all -- ctags_reader.py already documents CTAGS_CLASS_KINDS["haskell"] = set() on purpose, no class-shaped ctags kind exists for this language. The remaining 2 unvalidated haskell shapes (ctags-alone existence, ctags-misses-real existence) are still open -- under investigation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
agree[ctags]_vs[gitgalaxy,tree_sitter] (103 occurrences): dispatched agent investigation, all 10 sampled cases are ctags-side artifacts, no GitGalaxy/tree-sitter defect. Three causes: multi-clause double-tagging (ctags tags every pattern-match equation as a separate occurrence, GG/TS correctly anchor to the first clause only), keyword-as-identifier misparsing (class/where/pattern), and a value-vs-function kind collapse ctags' Haskell parser can't express (GitGalaxy and tree-sitter's own audit tooling both independently make the same correct distinction, per #1312/#1566). Full evidence in the ledger verdict. 5 of 8 haskell shapes now validated. 1 remaining under investigation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
agree[gitgalaxy,tree_sitter]_vs[ctags] (69 occurrences): dispatched agent investigation confirms all 10 sampled misses -- and by cross-check against further non-sampled instances, plausibly all 69 -- are locally-scoped definitions (instance methods, where-clause helpers, let-bound names in do blocks), never top-level. ctags' Haskell parser has no layout-rule/scope awareness and only tags column-1 equations; it correctly handles multi-clause definitions when they're top-level (expandFilterPath/writeFnBinary/writerFn all tag fine), so this is a pure scope gap distinct from the clause-splitting bug fixed earlier. GitGalaxy and tree-sitter both correct; nothing to fix in this repo. Added a doc addendum to ctags_reader.py alongside its existing Haskell notes so this is documented at the source, not just in the ledger. All 8 haskell ledger shapes are now resolved: 5 validated (real findings, all confirming GitGalaxy/tree-sitter correctness against ctags-side limitations, plus one genuine GitGalaxy false-positive and one documented point-free-arity distinction), 3 stale/no-longer- reproducing (artifacts of the clause-splitting bug fixed earlier in this PR, kept as history per the ledger's own lifecycle). Regenerated chart.svg reflects the cleared asterisks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
recall_t = matched / union-of-everyone's-claims looked principled but shares the exact weakness any single tool's own noise: this session found ctags multi-clause double-tagging (haskell) and, before that, a tree-sitter clause-splitting bug that both inflate the SHARED union denominator, silently deflating every OTHER tool's recall percentage in the process. A ranking built on a denominator any one tool can corrupt isn't a ranking worth showing. Func Recall / Class Recall panels are now Functions Found / Classes Found: each tool's raw claim count, no ratio, no winner badge, excluded from the summary tally -- "found more" isn't a correctness claim (a hallucinating tool finds more too). Precision panels are unchanged and still meaningful: their denominator is each tool's OWN claim count, never a cross-tool union, so it can't be corrupted the same way -- this is already the "of what we found, how many are real" framing. No changes to tri_comparison_reconcile.py/ledger.py -- the underlying per-slot agreement data was already computed this way; only how the chart renders two of the five panels changed. Regenerated across all 45 languages; confirmed Haskell's ctags multi-clause inflation (148 GG / 146 tree-sitter / 180 ctags) now renders as a plain, unranked fact instead of implying a winner. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Args' matched_consensus already meant "matched the cross-tool majority vote" -- exactly the same unverified-agreement-as-correctness assumption just removed from Func/Class Recall, just expressed as a 2-vs-1 majority credit instead of a union denominator. Either shape lets one tool's own noise (ctags' Haskell multi-clause double-tagging, tree-sitter's now-fixed clause-splitting bug) distort another tool's score. Args Match -> Args Found: each tool's raw comparable-args count (MetricScore.total_slots, not matched_consensus -- matched_consensus would still be the majority-vote correctness claim we're trying not to show pre-verification), no ratio, no badge, doesn't enter the summary tally. Same treatment as Functions/Classes Found, using a different underlying field since args' "found" concept isn't gate-free the way func/class existence is (args' total_slots still requires >=2 tools to have a comparable value -- an honest, noted limitation, not a silent one). Once a language/metric's ledger shapes are validated, a real denominator can come back for that language -- this is deliberately temporary, not a permanent downgrade. Summary tally is now Func/Class Precision only (2 ranked panels, down from 3). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real problem found on the regenerated chart: a strictly-highest-rate_pct
badge rule doesn't know or care about sample size or whether anyone has
actually checked the disagreement against source -- a 2-sample cell at
100% (2/2) could badge over an 80-sample cell at 98.75% (79/80) with
equal visual confidence. That's not a tool being more correct, it's an
artifact of an unverified comparison.
Replaces is_language_metric_clean() (GitGalaxy-only, aspect-specific)
with has_open_question() (tri_comparison_ledger.py): symmetric across
all three tools, one question -- does ANY currently-reproducing,
unvalidated ledger entry exist for this (language, symbol_type, metric)
triple, regardless of which tool(s) are on which side. Drives two
chart behaviors, both now symmetric instead of GitGalaxy-only:
- EVERY tool's value label in a disputed cell gets `*`, not just
GitGalaxy's -- the open question is "has anyone verified this",
not "did GitGalaxy specifically lose".
- No winner badge is drawn on a disputed cell at all, on any panel,
regardless of how lopsided the raw numbers look.
Confirmed working as intended on the regenerated chart: haskell (5 of
8 ledger shapes validated this session) now shows zero asterisks and a
real badge on Func Precision; c and rust, never investigated, are
honestly asterisked across every panel with no badges at all. Earning
a badge now requires the same verification work already used for
haskell -- reading real source and recording a verdict in the ledger
-- not just a bigger number.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-fix # Conflicts: # docs/self_scan/tri_comparison_chart.svg # docs/self_scan/tri_comparison_ledger.json
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1865 (already merged). PR #1865's branch was reused for this work since it built directly on that fix; #1865 itself is unaffected -- this PR is just the 2 commits on top of it that hadn't merged yet.
Functions Found/Classes Found(raw per-tool counts, no denominator, no winner badge, excluded from the summary tally). The oldmatched / union-of-everyone's-claimsdenominator sounded principled but turned out exactly as corruptible as any single tool's own noise -- confirmed twice on real data this session: ctags' Haskell parser double/triple-tags multi-clause functions, and tree-sitter's own raw walk had the identical clause-splitting bug (fixed in Fix haskell clause-splitting in tri-comparison's tree-sitter walk #1865) before that. Either bug inflates the shared denominator, silently deflating every other tool's recall percentage too.matched_consensusalready meant "matched the cross-tool majority vote," the same unverified-agreement-as-correctness assumption, just expressed as a 2-vs-1 majority credit instead of a union denominator. NowArgs Found: each tool's raw comparable-args count, no ratio, no badge.tri_comparison_reconcile.py/tri_comparison_ledger.py-- purely a chart-rendering change; the underlying per-slot agreement data was already computed this way.This is explicitly framed as temporary in the code comments: once a language/metric's ledger shapes are validated (per
docs/self_scan/how_to_investigate_a_discrepancy.md), a real denominator can come back for that language specifically.Verification
--languages haskell,rustbefore the full run and visually confirmed via PNG export: found-count panels show plain numbers with no badges (e.g. haskell Functions Found: GitGalaxy=148, tree-sitter=146, ctags=180 -- the ctags multi-clause inflation is now a visible fact instead of implying a winner); precision/asterisk behavior unchanged.tri_comparison_chart.py --all --writeacross all 45 languages.ruff_audit.py --ci/mypy_audit.py --ci: no new findings beyond baseline.Test plan
--all --writeregeneration, diffed against prior state.ruff_audit.py --ci/mypy_audit.py --ciclean.🤖 Generated with Claude Code