fix: adjudicate distinguishes part-whole from identity and stops showing a flat confidence#147
Merged
Merged
Conversation
…ing a flat confidence The adjudication rubric now states SAME means the same entity under different names (synonyms/aliases) and that a part, component, aspect, subtype, instance, or example of X is a DIFFERENT entity, biasing toward different/uncertain for part-whole pairs since SAME feeds a destructive merge. Verified live on qwen3:8b: the MCP/MCP-Server part-whole pair now returns DIFFERENT 3/3 (was SAME), while a true MCP/Model-Context-Protocol synonym pair still returns SAME 3/3. The per-verdict confidence number is no longer displayed: a local model returns a flat, uncalibrated value (a constant 0.95), so a two-decimal number implied a precision it does not have. The value is still parsed and kept on AdjudicatedCandidate for future thresholding. Closes #138
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
openkos adjudicatehad two defects (issue #138):SAME. In a real MCP run,mcp/mcp-server,mcp/mcp-primitives, etc. — all components of MCP — were labelledSAME, which feeds a destructivemergethat would collapse six distinct concepts into one.confidence: 0.95on every verdict. A constant, uncalibrated value shown to two decimals implies a precision that does not exist.Fix
resolution/adjudication.py):SAMEnow means the same entity under different names (synonyms, aliases, casing/spelling variants). A part, component, aspect, subtype, instance, or example of X is a DIFFERENT entity — "a component of X is not a duplicate of X" — and the model is told to preferdifferent/uncertainfor part-whole pairs becauseSAMEtriggers a destructive merge.cli/main.py): the confidence number is no longer rendered. The value is still parsed and kept onAdjudicatedCandidate.confidencefor future thresholding — it is just not shown as false precision.Empirical validation (live
qwen3:8b)The part-whole fix does not over-bias: genuine synonyms still resolve to
SAME. Confidence stayed flat at0.95in the replay, confirming why hiding it is the right call.Tests
test_system_prompt_distinguishes_part_whole_from_identitypins the rubric against regression.Closes #138
🤖 Generated with Claude Code