fix(claims): bind parity wording to benchmark data - #421
Merged
Conversation
… copy tests/aiDiscoverability.test.js asserted that public/llms-full.txt contains the words 'parity on success'. That pinned a sentence whose truth depends on data nothing was comparing it to. When the saved-row oracle re-adjudicated OpenEMR compiled run 20 and the arm became 19/20 against the agent's 10/10, the test went on requiring the older, friendlier claim. A test that enforces copy can enforce wrong copy. The replacement reads data/benchmark.json. Every arm's counts must appear on the page as a ratio or as its percentage. Where the two arms do not tie, any sentence claiming the comparisons show parity must name where parity fails. This is RED on purpose. The limitations bullet still reads 'The benchmark comparisons show parity on success', which the same file contradicts a few hundred lines earlier. Correcting that sentence is a prose change owned by the correction pass, not by this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
✅ Deploy Preview for cosmic-klepon-3c693c canceled.
|
abrichr
marked this pull request as ready for review
August 28, 2026 20:01
abrichr
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
public/llms-full.txtnow states the two observed outcomes. MockMed tied on success. In OpenEMR, the agent arm recorded 10/10 and compiled replay recorded 19/20. The copy says these unequal, small samples don't establish superiority on accuracy.The test reads
data/benchmark.json, requires each arm's measured ratio or percentage, and rejects an unqualified parity sentence when any comparison doesn't tie.Why
The old bullet said both benchmark comparisons showed parity. It contradicted the OpenEMR table and artifact after the saved-row oracle changed the compiled result to 19/20.
The old assertion required only the words
parity on success. That assertion could keep a wrong claim green. The new assertion derives its expectation from the artifact.Review
Check that the copy matches
data/benchmark.jsonand that a future artifact change makes the test reject stale parity wording.Verification
node --test tests/aiDiscoverability.test.js(22 passed)npm run check:figuresnpm run check:universalsnpm test(288 passed)npm run build(50 pages generated)