Skip to content

v1.108.140 — the first gold corpus: channel accuracy, measured

Choose a tag to compare

@jgravelle jgravelle released this 18 Jul 15:04

Added

  • An authored gold corpus for find_implementations channel accuracy (benchmarks/goldset/). Every implementation relation — declared subclasses, duck-typed conformers, decorator-registered handlers — and every deliberate false-positive trap (a module-homonym base class, same-name-different-domain methods like electrical/legal charge, substring decorator matches) is labeled in gold.json with a per-pair rationale, so ground truth is exact by construction.
  • A reproducible measurement harness (benchmarks/goldset/measure.py) writing per-channel precision/recall to benchmarks/provenance/channel_accuracy.json. First measured round: AST 0.833 precision, duck 0.6, decorator 0.6 — recall 1.0 across all channels. The shipped priors survive contact with measurement: 0.85 declared vs 0.833 measured for AST, 0.65 vs 0.6 for duck, and the 0.45 decorator prior is more conservative than measured.
  • The measurement re-runs in CI: the committed artifact must equal a live re-measurement and the corpus content hash — the numbers cannot drift from the reproducible run.
  • Registry and responses carry the measured reference. Each heuristic channel's provenance entry gains measured_ref beside its declared ranking prior, surfaced in _meta.confidence_provenance. Operating constants deliberately stay declared: recalibrating ranking to a small-n corpus would trade honest stability for false precision, and the artifact states its scope (authored-pattern discrimination, not in-the-wild base rates).