fix(#343): regenerate NIH referee transcripts post-#338, recover full n=35 - #359
Merged
Conversation
sebasmos
added a commit
that referenced
this pull request
Jul 29, 2026
…r, and real tests
Three divergences from the ad-hoc parser this PR replaces, all found by restoring the frozen real-cache
fixture the PR deleted. 9 of its 21 cases were failing.
1. Full-text options fell through to the trailing-letter scan on multi-match, so a bare "A" in the prose,
almost always the English article, became a confident vote for option A. "Both Type II pneumocytes and
Club cells are plausible. A surfactant deficiency argues for the former." returned option A instead of
Club cells. The non-letter branch is now terminal on match and keeps the old last-mentioned behaviour.
One deliberate exception: an explicit disjunction such as "yes or no" abstains, which is what this
module's own yes/no/maybe goldens require and which the old parser did not do.
2. The boxed declaration pattern had no guard after the captured letter, so
\boxed{\text{Anti-mitochondrial antibodies}} captured the "A" of "Anti" and scored a full-text answer
as option A. Added (?![A-Za-z]). The bare \boxed{C} and \boxed{\text{C. ...}} forms still match.
3. The "answer is" patterns did not allow the emphasis markers the models actually emit, so
"The correct answer is **D**." parsed as unparseable. The old parser allowed \** here.
Tests: tests/test_parse_regression.py had no def test_ at all, so pytest collected zero cases from it while
this PR also deleted tests/test_reproduce_parse_choice.py (5 tests) and its fixture. Net was 5 tests removed
and 0 added, on the parser every MedQA and imaging number depends on. Restored the fixture and wrote 40 real
cases: the three multi-match rows, a parity sweep against an inlined copy of the pre-centralization parser,
all 21 frozen real-cache replies, and the anti-gutting guard whose docstring on main already named this
exact failure mode.
Proved non-vacuous: with the pre-fix extract.py restored, 16 of the 40 fail.
763 passed, 7 skipped, ruff clean. No committed result file is touched, so no reported number moves.
Still behind main by the #350, #358 and #359 test additions; a rebase will pick those up.
sebasmos
added a commit
that referenced
this pull request
Aug 4, 2026
… n=35 (#359) Both imaging_referee.jsonl and imaging_judge_referee.jsonl predated the #338 plant-direction fix, so 13 of their 35 rows described committee boards the corrected design no longer produces. The workaround was referee_valid_subgroup.py, restricting every citable referee figure to the 22 rows both designs agree on. That workaround is no longer needed. imaging_referee.py does not plant anything itself: it reads rows from imaging_cascade.py and its only new call is an independent re-read keyed on the finding, not on the plant. The cascade transcript was already post-fix. So re-running the referee against it regenerates all 35 rows correctly, and because the re-read prompt does not depend on the plant direction every one of them was already in the committed cache. Cost: 0 new calls for the referee, 2 for the judge. Full cohort, n=35: detector P R FPR naive gate 0.559 1.0 0.9375 same-lineage judge 0.559 1.0 0.9375 deployable referee 0.882 0.789 0.125 Two things get stronger, not weaker. The judge is now identical to the naive gate on all three metrics across the whole cohort rather than a subgroup, so "the judge collapses to the gate in imaging" is exact. And referee_vs_naive improves materially: risk difference 0.26 with a bootstrap interval of [0.029, 0.486] that EXCLUDES 0, where the pre-fix file gave 0.17 with [-0.029, 0.343] straddling it. Power 0.59, McNemar b=13/c=4. That comparison was previously reported as directionally suggestive but not powered. Regenerated downstream: imaging_referee_agreement.json, panel_vote.json, effect_sizes_imaging.json. Removed the now-false CAVEAT that effect_sizes_imaging.py hardcoded about the pre-fix transcript, and corrected its read prose, which still quoted the old 0.17 / b=9 / c=3 / power 0.41 figures. Same stale-hardcoded-prose bug class that #350 fixed one field over. README referee table and the ground-truth count (22 to 19 of 35) updated. referee_valid_subgroup.py now reports 35 of 35 valid and 0 excluded, so it degenerates to a guard rather than a workaround. Its three tests were pinning the 22-row restriction, so they now pin the real invariant: no row in either transcript may plant against the model's clean read again. 798 pass, ruff clean, recompute_derived --check clean.
sebasmos
added a commit
that referenced
this pull request
Aug 4, 2026
…r, and real tests
Three divergences from the ad-hoc parser this PR replaces, all found by restoring the frozen real-cache
fixture the PR deleted. 9 of its 21 cases were failing.
1. Full-text options fell through to the trailing-letter scan on multi-match, so a bare "A" in the prose,
almost always the English article, became a confident vote for option A. "Both Type II pneumocytes and
Club cells are plausible. A surfactant deficiency argues for the former." returned option A instead of
Club cells. The non-letter branch is now terminal on match and keeps the old last-mentioned behaviour.
One deliberate exception: an explicit disjunction such as "yes or no" abstains, which is what this
module's own yes/no/maybe goldens require and which the old parser did not do.
2. The boxed declaration pattern had no guard after the captured letter, so
\boxed{\text{Anti-mitochondrial antibodies}} captured the "A" of "Anti" and scored a full-text answer
as option A. Added (?![A-Za-z]). The bare \boxed{C} and \boxed{\text{C. ...}} forms still match.
3. The "answer is" patterns did not allow the emphasis markers the models actually emit, so
"The correct answer is **D**." parsed as unparseable. The old parser allowed \** here.
Tests: tests/test_parse_regression.py had no def test_ at all, so pytest collected zero cases from it while
this PR also deleted tests/test_reproduce_parse_choice.py (5 tests) and its fixture. Net was 5 tests removed
and 0 added, on the parser every MedQA and imaging number depends on. Restored the fixture and wrote 40 real
cases: the three multi-match rows, a parity sweep against an inlined copy of the pre-centralization parser,
all 21 frozen real-cache replies, and the anti-gutting guard whose docstring on main already named this
exact failure mode.
Proved non-vacuous: with the pre-fix extract.py restored, 16 of the 40 fail.
763 passed, 7 skipped, ruff clean. No committed result file is touched, so no reported number moves.
Still behind main by the #350, #358 and #359 test additions; a rebase will pick those up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the real half of #343 for the NIH lane: the two referee transcripts that predated #338 are regenerated, so the 22-row valid-subgroup workaround is retired and the full n=35 is citable again.
Why this needed almost no API budget:
imaging_referee.pyplants nothing itself. It reads rows fromimaging_cascade.py, which was already post-fix, and its only new call is an independent re-read keyed on the finding rather than on the plant direction. So every referee re-read was already in the committed cache. 0 new calls for the referee, 2 for the judge.Full cohort, n=35:
Two results get stronger:
referee_vs_naiveimproves materially: risk difference 0.26 with bootstrap [0.029, 0.486], which excludes 0. The pre-fix file gave 0.17 with [-0.029, 0.343], straddling it, and was written up as "directionally suggestive but not yet powered to rule out chance". Power 0.59, McNemar b=13/c=4.Also fixed two stale-prose defects of the same class #350 dealt with:
effect_sizes_imaging.pyhardcoded a CAVEAT saying the referee transcript predates #338, and itsreadstring still quoted 0.17 / b=9 / c=3 / power 0.41 after the numbers moved.referee_valid_subgroup.pynow reports 35 of 35 valid, 0 excluded, so it becomes a regression guard rather than a workaround. Its three tests were pinning the 22-row restriction; they now pin the invariant that no row may plant against the clean read again.798 pass, ruff clean,
recompute_derived --checkclean.