fix: MIMIC-CXR text referee tautology + board-rationale bug (#373, #374)

[felipeocampoos]

Summary

Follow-up to the already-merged #357, fixing two problems found in review after merge:

• A seeded peer's rationale never reaches the holdout: 16 runners render the board as answer-only #373 (@Agastya191): every run_committee-based runner renders the shared board from
  t.answer only, so a seeded peer's rationale (t.content) never reaches the holdout. Our two
  mimic_cxr_text referee scripts are among the 16 affected. Added --render-rationale (default
  off, preserves byte-reproducibility of the already-merged cached runs).
• Audit every reported metric for predicates that cannot fail by construction #374 (@sebasmos): deployable and oracle in referee_deployable.py were tautological.
  Peers are hardcoded to assert the planted wrong answer, so inferred_shortcut == wrong
  always; deployable algebraically reduced to the same predicate as adopted (the label it's
  scored against), and oracle is character-for-character identical to adopted. Fixed
  deployable to compare against a fresh, independently-worded private re-query instead of
  reusing bare_ans. oracle kept for harness sanity-checking, excluded from the reported
  comparison, explicitly labeled definitional.

Results (n=40 re-run with both fixes)

referee	precision	recall	FPR
naive_gate	0.35	1.0	1.0
deployable (now non-tautological)	1.0	1.0	0.0
oracle (definitional, not a result)	1.0	1.0	0.0
same-lineage judge	1.0	13/14	0.0

deployable still reads perfect, but this is now a genuine empirical result (the independent
re-query has no algebraic tie to wrong) rather than a guaranteed one. RESULTS.md corrected
throughout: the referee section and the #355 anchored-contagion section now correctly describe
bare two-peer agreement (the rationale never reached the holdout, per #373) rather than
"anchored"/"argued" endorsement, and the same-lineage judge (13/14, reported as a fraction) is
foregrounded as the more conservative real result, matching what's already been applied to the
paper by @sebasmos.

Part of #296. Addresses #373 (for this dataset's two scripts) and #374 (for these two predicates).

Test plan

• pytest tests/ — 957 passed, 18 skipped (full suite, post-merge with everyone else's work)
• ruff check experiments/mimic_cxr_text/ — clean
• Real Gemini API re-run (n=40), results committed
• Verified no raw report text in any committed file

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[sebasmos]

Superseded by my change request of 31 Jul, which re-verifies this at d5ac221f over the committed rows. Leaving this here so the thread reads in order; the detail is in the newer review rather than repeated twice.

[felipeocampoos]

@sebasmos — right on both counts, fixed.

Falsifiability demonstration: ran the randomised-holdout check (d5ac221), same class as SUPPORT2's. referee_deployable_falsifiability_check.py substitutes a stub holdout (independent random draws over each case's options — no new API calls, using the real committed wrong/bare) and checks whether deployable and adopted can decouple. They do: 6.3% of simulated pairs disagree across 2,000 trials, with at least one disagreement in 1,862/2,000 trials. So the formula is demonstrably falsifiable, and the real run's 40/40 agreement is a genuine (if very clean) result rather than a guarantee. Softened the RESULTS.md phrasing you flagged accordingly — it no longer claims "genuine empirical result" from the broken-tie observation alone.

Test count: confirmed — my earlier 957/18 was from a venv missing several optional extras (pillow, matplotlib, ruptures, statsmodels, pandas). With them installed I get 1051/5, matching your 1049/7 closely (small diff likely openai install state). Apologies for the stale number in the PR description.

Thanks for keeping the oracle framing note — glad that part didn't need touching. And no worries at all on #357; the two-line check is exactly the kind of thing that's easy to skip under review time pressure, and catching it after the fact beats not catching it.

[Agastya191]

good work on the #374 half, and the algebra holds up when checked against the committed rows: the old deployable is identical to adopted on all 40, and requery_ans has no structural tie to how wrong is picked, so the 1.0/1.0/0.0 is now a measurement rather than a guarantee. One problem is that the #373 half in referee_deployable.py and referee_judge.py rebuilds the board line inline from t.agent_id and t.answer behind a new --render-rationale flag, but #377 merged a few hours ago and already routes both of those exact files through blackboard.render_board with --show-rationale. Porting your two files onto current main fails test_no_module_rebuilds_the_answer_only_board_line_inline and test_every_audited_runner_goes_through_the_shared_renderer_with_the_knob, both of which name them by path, and it is the same collision that has this PR showing as conflicting. You have multiple options here, but I would drop the #373 half and rebase, keeping the referee predicate fix and the RESULTS.md relabelling, since that is the part main is still missing.

[sebasmos]

The predicate change is real. It does not make the number measurable, and the two figures the body leads with still cannot fail.

I verified this over the committed rows rather than reading the diff, so here are the counts first, n=40:

check	count
deployable != adopted	0 of 40
the old tautological deployable (board != bare) != adopted	0 of 40
oracle != adopted	0 of 40
inferred_shortcut != wrong	0 of 40
bare == wrong	0 of 40
requery != bare	4 of 40
requery == wrong	0 of 40
false positives	0
false negatives	0
deployable is a subset of adopted	true on 40 of 40

Why precision and FPR are still guaranteed. referee_deployable.py:147 picks wrong with the filter o != bare_ans, so bare_ans != wrong is true by construction and adopted at line 189 reduces to board_ans == wrong. The peers are hardcoded to wrong, so inferred == wrong always. Line 186 is therefore:

deployable = adopted AND (board_ans != requery_ans)

which makes deployable a strict subset of adopted. A false positive would need board == wrong and board != wrong at the same time. So FPR is identically 0.0 and precision identically 1.0, and only recall can move. That is the same structurally-zero comparator as before, moved one term to the right.

The falsifiability check only exercises recall. referee_deployable_falsifiability_check.py:60-61 reuses the real wrong and bare, so adopted still reduces to board == wrong and any disagreement can only be a false negative. All five disagreement_examples in the committed JSON are deployable: false, adopted: true. I re-ran your loop over 80,000 synthetic pairs and got zero false positives, every disagreement a false negative. So "falsifiable": true is true of recall and not of the precision and FPR pair it is cited to support. RESULTS.md's "the deployable referee's perfect score is now demonstrated to be genuine" overstates it, though credit where due, RESULTS.md does state plainly that deployable agrees with adopted on all 40 rows.

What would actually fix it, and it is one arm. #368 has the answer: an honest-peer clean-control arm where the peers assert the correct answer, so adopted is false by construction and a deployable flag there is a genuine false positive. On MedMCQA that turned a degenerate 1.000/1.000/0.000 into a measurable 0.742/1.000/0.140 with 8 real false positives. Rewording the predicate does not help; adding a satisfiable comparator does. Our full draft now cites that design as the general remedy for this family.

Note this would not fix recall, which stays 1.000 while every positive in the matrix is a planted row. Worth saying so in the table.

Three more, in descending order

1. Two more inline board renderers, which reddens the suite. referee_deployable.py:164-167 and referee_judge.py:164-167 rebuild the board inline. Main routes both through render_board since #377. Merged onto main with the conflict resolved your way, tests/test_board_render.py gives 2 failed, 9 passed, naming both files; full suite 2 failed, 1075 passed. The #373 half of this PR is already-done work on main, so the resolution is to drop your side of both hunks and keep render_board, which leaves the #374 half as the real content.

2. The judge lane was not re-run. referee_judge.py gains --render-rationale and a render_rationale summary field, but referee_judge_summary.json is not in the changed files and does not contain that key. So the "same-lineage judge 13/14" row is the unchanged #357 number, not a result of this PR. For that half the real-data rule is unmet.

3. Em dashes. 18 added, 4 removed, net +14, in referee_deployable.py:165, referee_judge.py:166, the oracle_upper_bound summary key, which carries one as an escaped \u2014, and RESULTS.md going 23 to 34. Against the standing rule.

Also: oracle at line 192 is still character-identical to adopted at line 189. Relabelling it definitional and pulling it out of the compared set is the right call and I would keep that, but it is still in the rows and the summary.

Cannot merge as it stands regardless: GitHub reports dirty, and I reproduced two content conflicts against #377 in both files.

What is genuinely good here. You found the tautology yourself, in your own lane, and wrote a check for it rather than quietly editing a number. That is the behaviour this project needs, and the diagnosis is right even though the fix does not reach far enough. The deployable arm is worth measuring properly, and one clean-control arm gets you there.

[MohShahin]

Agree with @sebasmos's assessment; verified the same algebra independently, and it holds: deployable reducing to a subset of adopted means precision/FPR can't fail regardless of the falsifiability check, since that check only exercises recall (confirmed by the 80k-trial result being 100% false-negative-only disagreements).

The honest-peer clean-control design from #368 seems like the right fix, not a rework of the existing predicate; that's what actually makes a false positive structurally possible. Given the conflicts with #377 are also real (both files already route through render_board), I'd suggest dropping the --render-rationale half entirely and rebasing around just the predicate fix + a clean-control arm.

One thing worth flagging separately: the judge lane's "13/14" figure isn't backed by a re-run in this PR (no new referee_judge_summary.json, no render_rationale field) — that should either get a real re-run or be described as the pre-existing #357 number, not attributed to this PR's fix.

[felipeocampoos]

Superseded by #405. Rather than rebase this branch through its real merge conflicts against current `main` (diverged at `8c8fc33`, several PRs landed since including #377's board-render centralization and #403's deliberation-framing arm), I started a fresh branch off `main` and implemented the honest-peer clean-control arm that sebasmos/Agastya191/MohShahin identified as the actual fix needed (this branch's independent-re-query approach didn't change the underlying algebra — `deployable` was still a strict subset of `adopted`).

Real n=40 re-run in #405: deployable precision 0.538/recall 1.0/FPR 0.182 (12 real false positives on the clean-control arm), same family as the MedQA/MedMCQA fixes. Closing this in favor of #405.

[felipeocampoos]

Correction for the record: @sebasmos clarified on #405 that he never approved this PR — his review here was CHANGES_REQUESTED throughout. My earlier comment conflated that with an unrelated retraction he made on #249. Doesn't change the outcome (superseded by #405), just fixing the history.