feat(match): confidence calibration, honesty metrics, disambiguation (D2/D6/G1) — Gate 4#30
Merged
Conversation
…ation (D2/D6/G1) — Gate 4 Close out the matching engine (TRACKER step 4.5): - Calibration: confidence thresholds live in core (CONFIDENCE_THRESHOLDS); a new `node eval/dist/run.js --calibrate` measures precision at the `high` cutoff on the eval set and records eval/calibration.json (precision@high 1.000, n=71). - Honesty metrics + gates: the scorecard now reports highConfidenceCorrect, ambiguityHonesty, and poisonRate; thresholds.json gates them (=1.0 / ≥0.90 / ≤0.05). Current run: 1.000 / 1.000 / 0.000. - Ambiguity protocol (D6/G1): the `disambiguation` question is now built from the DIFFERENCES between the tied leaders — "Which one — BillingForm (shows 'Card number'), or ProfileForm (shows 'Display name')?" — so the caller can answer with a resolving term. Declined reasons stay machine-readable (no-signal / not-our-app / …). 40 core + 3 vision + 102 parser tests pass; eval green with the new gates. **Gate 4 passes — matching engine complete (M4).** Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 4.5 — Confidence calibration & ambiguity protocol (closes Gate 4)
Failure modes D2 / D6 / G1. The final matching-engine step.
CONFIDENCE_THRESHOLDS). Newnode eval/dist/run.js --calibratemeasures precision at thehighcutoff on the eval set and writeseval/calibration.json(precision@high 1.000, n=71).highConfidenceCorrect,ambiguityHonesty, andpoisonRate;thresholds.jsongates them at = 1.0 / ≥ 0.90 / ≤ 0.05. This run: 1.000 / 1.000 / 0.000.disambiguationquestion is built from the differences between the tied leaders:Verification
pnpm eval: green with the new calibration/honesty/poison gates.Gate 4 passes — the matching engine (Phase 4) is complete (milestone M4: screenshot/text → ranked, calibrated, honest matches).
🤖 Generated with Claude Code