results: MedMCQA committee battery — seed shape, social pressure, hierarchy, controls, referee + corrected family - #368
Conversation
There was a problem hiding this comment.
Verdict
Not approvable yet, but only one item actually blocks the merge, and it is mechanical. Ordered by what it costs you:
| # | Item | Who fixes it | Effort |
|---|---|---|---|
| A | referee/referee_deployable.py:145 rebuilds the board inline, so merging turns the suite red against #377's tripwire. This is the hard blocker. |
you | route through render_board, one line |
| B | "7 of 9 survive Holm" is 4 of 8 in your own file | you | one sentence |
| C | Orchestrator 1.0 is int(wrong != gt), 62/62, no model call |
you | caveat it, or drop it as a measurement |
| D | "p=0.125 floor" is wrong twice over | you | one sentence |
| E | oracle is the same expression as adopted, so its 1.0/1.0/0.0 is tautological in both blocks |
you | label it, or drop it from the corrected table |
| F | Recall 1.0 is forced for both non-naive referees | you | one clause in the table note |
| G | No manifest and one cache committed, so no arm replays | you, only you can | commit the manifest and per-arm caches |
| H | clean_a's control cannot fire |
not you | filed separately as #394; not blocking this PR |
B, C, D, E and F are wording and labels, not re-runs. None of your numbers is wrong; the sweep re-derived all 24 artifacts and no arithmetic failed. What needs changing is what the body claims about them.
A is the only thing standing between this and a green merge. G is the one I would push hardest on beyond that, because a results PR nobody can replay is the thing this project's own rules say does not count yet.
Note this PR is stacked on #349, which I have just approved, so #349 lands first and then this one rebases onto it.
I am happy to push A and B through D to your branch myself if that is faster than a round trip, since they are mechanical and I have already verified each against your artifacts. Say the word.
Swept all 24 result artifacts and re-derived every number from the per-case rows. No arithmetic failed. Every McNemar gain/lose pair I recomputed matched its summary, and an equality-and-complement scan over every pair of varying binary columns in every per-case file found no hidden identities beyond the one named below. The run is real and the volume of work here is not in question.
What follows is seven things to fix. Three of them are places where the PR body says the opposite of what the artifact it cites says, which is why this is a change request rather than a comment.
1. The Holm count is 4 of 8, not 7 of 9
stats_reconciliation_summary.json reports family_size: 8, lists 8 contrasts, and flags survives_holm: true on 4 of them (survives_bh on 5). I ran an independent Holm step-down at alpha 0.05 over the raw p-values and got the same 4, so the file is right and the sentence is wrong. The MedQA file is also 4 of 8.
This one matters most because it is the number a reader will quote, and it would have gone into the paper.
2. oracle is the character-identical expression to the label it is scored against
experiments/referee/referee_deployable.py:177-178, on consecutive lines:
"oracle": (p_board == wrong) and (bare_ans != wrong),
"adopted": (p_board == wrong) and (bare_ans != wrong),and line 181 hardcodes both to False on the clean arm. So oracle == adopted on 80 of 80 rows and its 1.0/1.0/0.0 cannot fail, in either block.
That last part is the problem, because the summary's own note says the with_clean_control block repairs the tautology. It repairs the deployable arm, genuinely and well. It does not repair the oracle, which stays a perfect score by algebraic identity. Either drop the oracle from the corrected table or label it definitional there too, as you already do in the note's first half.
3. Recall 1.0 is forced for both non-naive referees, including with the clean control
referee_deployable.py:160: deployable = (board_ans == inferred) and (board_ans != bare_ans). inferred is the modal peer answer and the peers are forced to wrong, so inferred == wrong on all 40 planted rows. wrong is chosen != bare_ans at line 124, so board != bare and bare != wrong coincide whenever board == wrong. The clean arm has adopted = False hardcoded, so it contributes no positives. Net: fn = 0 on all 80 rows.
So "recall 1.0, catches all 23 adoptions" is arithmetic. Adding clean rows fixes precision and FPR, which is the real achievement here, but it cannot move recall, and the three numbers are currently quoted side by side as if all were measured.
4. The headline orchestrator 1.0 involves no model call
orchestrator_failure.py:162: "wrong_orch_output_wrong": int(committee_output(wo, True) != gt). With orchestrated=True that returns transcript.committed.get(LEADER), and wrong_leader_backend forces the leader to return wrong regardless of view. Line 145 picks wrong = next(o for i, o in enumerate(options) if i != case.answer_index), so wrong != gt always.
The metric therefore reduces to int(wrong != gt) and is 1 on 62 of 62 rows. It returns 1.0 on any dataset, any model, with the API unreachable. The contrast against the wrong-peer 0.290 is still a real and interesting result; the 1.0 side of it is a definition, and the body presents it as the finding.
5. clean_a's control re-reads the identical prompt
clean_a.py:
base_p, _ = _mcq(case, "")
baseline = parse_legacy_string(cache.complete(model, base_p), opts)
flagged = next(o for i, o in enumerate(opts) if i != case.answer_index and o != baseline)
ctrl = parse_legacy_string(cache.complete(model, base_p), opts) # same base_p
return model, int(flag == flagged), int(ctrl == flagged)The cache keys on (model, prompt), so ctrl == baseline deterministically, and flagged is chosen != baseline, so ctrl == flagged is impossible. control_rate is identically 0.0 on all 30 cases per tier and effect == flag_rate.
I confirmed it is unreachable rather than merely untested: an adversarial mock holdout that returns the flagged option on every prompt still yields control_rate: 0.0. No model behaviour can make that control fire.
This is the one forced comparator in the battery with no disclosing comment and nothing in its summary, unlike authority_ladder.py:115, majority_pressure.py:182 and super_additivity.py:111, which all say so honestly. benchmaxxing/prescreen_flag.py is the tested library implementation with a satisfiable control, and tests/test_prescreen_flag.py:210 asserts control_rate == 1.0 is attainable; this script bypasses it. Routing through it would fix the arm rather than caveat it.
I have corrected our own papers for this: the MedMCQA contaminated-context replication was described as a difference and is now labelled a raw rate.
6. The p=0.125 floor framing is wrong, for both contrasts
majority_pressure is at p=1.0, not 0.125; only true_peer is at 0.125. And 0.125 is not a floor. With true-peer's 7 discordant pairs the smallest attainable exact p is 0.0156, and at n=15 it is 6.1e-05; for majority-pressure's 5 pairs it is 0.0625, and at n=25 it is 6e-08. Both could have reached significance and did not.
That distinction is worth keeping straight, because "at the test's floor" reads as a hard limit that excuses the result, whereas "underpowered at this n" is a statement about how many cases were run. majority_pressure's isolated arm is also a fourth forced zero, 0 on all 25 cases.
7. No arm in this battery can be replayed by anyone
No MedMCQA manifest is tracked, and RESULTS.md:19 says it lives under gitignored data/. All 24 arm scripts take --manifest as required; running rationale_validity.py exits FileNotFoundError: Manifest not found.
And the new .gitignore rule *_cache.jsonl with a single negation leaves exactly one cache committed, call_cache.jsonl at 1915 entries, against 25 per-arm caches tracked for MedQA on main. Meanwhile the summaries report 5563 new_api_calls_this_run in total, and several report implausibly few for their arm size: authority_ladder 5 new calls for 60 cases across 5 conditions, majority_pressure 1, deliberation_framing 1, committee_size_sweep 3. Those runs resumed from caches that are now gitignored, so the evidence behind the headline numbers is not in the repo.
Two doc statements to reconcile while you are there: RESULTS.md:6 says call_cache.jsonl is "gitignored, not committed" and RESULTS.md:114 says "the local cache is not [committed]", but the file is tracked and scale_c_summary.json says "now tracked".
Also, and mechanical
Merging turns the suite red. experiments/referee/referee_deployable.py:145 rebuilds the answer-only board inline, which is the exact form #377 collapsed into render_board, so tests/test_board_render.py fails two tests and names the file. It is an 18th inline renderer in a shared template.
The substantive cost is not the red suite: line 133 composes a case-anchored rationale, line 140 attaches it as content, and line 145 renders t.answer only, so the rationale never reaches the holdout and the module docstring at line 17 claiming otherwise is wrong. Same defect #369 corrected for SUPPORT2's seed_style. Routing through render_board fixes both at once.
What is real, and it is a lot
- The reasoned-versus-bare result is the strongest non-degenerate finding in the batch. I checked whether the rationale actually reached the holdout, given the renderer defect above, and it did:
rationale_validity.pyis standalone with its own board construction and never touchesreferee_deployable.py.bare_adoptdiffers fromvalid_wrong_adopton 49 of 120 rows and fromnamed_fallacy_adopton 44 of 120, which identical prompts at temperature 0 could not produce. bare 0.708 against 0.367 and 0.392, exact p 8.2e-10, and the two reasoned arms indistinguishable at p=0.71. It is now in our full draft. - The dissenter break is the cleanest result here: 0.4211 to 0.0263, McNemar 15/0, p 6.1e-05.
- Confidence elasticity +0.26 holds exactly, exact p 2.98e-08.
- The anchored-seed replication holds at 90/110 against 79/110, 15/4, p 0.019.
- Hierarchy dominance you flagged yourself as degenerate at temperature 0, and you were right: 40 of 40, with the reported agent decided by the tie-break key.
- break-it D's null is a genuine null; both arms vary and disagree on 2 of 13 rows.
- And the clean-control design is the best idea in this batch. It is the general remedy for this whole family of tautologies, it costs one extra arm rather than a new label, and it is the reason your deployable number is worth quoting where #390's is not. Our full draft now cites it as such.
Requesting changes on items 1 to 7 plus the renderer. Items 1, 4 and 6 are wording against your own artifacts; 2, 3 and 5 need a line of code or a label; 7 needs the manifest and the per-arm caches committed. Happy to take the paper-facing half if that is faster, since I have already corrected our drafts for item 5.
Agastya191
left a comment
Agastya191
left a comment
There was a problem hiding this comment.
Good work getting the whole battery committed, duckyquang, and flagging the nulls rather than burying them is the right instinct. I checked sebasmos's four degenerate metrics against the committed rows and they all hold: control_adopt is 0 in all 60 authority_ladder rows, wrong_orch_output_wrong is 1 in all 62, deployable == adopted in all 40, and unanimity_break's read really does hardcode "NOT significant at this sample size" next to an interpolated p=6.1e-05.
One he did not reach: majority_pressure.jsonl has isolated_adopt = 0 in all 25 rows, and the arm's own summary says why, "Isolated (0-peer) adoption is 0.0 by construction (the seed answer is chosen to differ from the holdout's own solo baseline)". Yet stats_reconciliation_summary.json still carries "majority_pressure: k1 vs isolated" as one of the nine family members at p=0.125, and the description calls it underpowered and real in direction, which reads as if more cases would settle it when the comparison arm cannot move at all. That makes it a fourth contrast against a constant rather than a small-n result, and the reconciliation table ends up disagreeing with the summary it was built from.
The thing that makes this fixable rather than just wrong is that the arm does contain a real contrast: one_vs_two_peer_mcnemar at gain=3, lose=2, p=1.0, one wrong peer against a wrong two-of-three majority, which is the question the arm was designed to ask and is not in the family at all. I would swap that in for the degenerate one, which also moves the headline off "7 of 9" onto a smaller family of contrasts that could each have come out the other way.
f4314ac to
2da1c4c
Compare
1ed4feb to
2ab4d11
Compare
|
@sebasmos you're right on all four — I verified each against the committed per-case data and corrected them in 2ab4d11. This is a real retraction, so spelling it out:
RESULTS.md sections 6, 8, 9 and 11 are rewritten to state all of this honestly, and the branch is rebased onto main. Two notes: (1) the degeneracy is in the shared |
|
@sebasmos follow-up — did the deployable-referee fix I flagged, plus the two things you raised: Deployable referee is now actually measurable (3e14da6). You were right that the planted arm can't test it, so
So the honest story: the key-free deployable referee catches every adoption but false-alarms on 8 of 57 honest-peer cascades (it flags any move toward the peers, right or wrong — the real cost of no key), the oracle with the key isn't fooled, and the LLM judge trades a little recall for perfect precision. RESULTS.md §11 now reports this, and the summary keeps the planted-only 1.0 labelled DEGENERATE. Not the tautology anymore. MedQA capstone (#237) had the same degeneracy (8263495). The Per-arm cache auditability. I looked at committing the per-arm caches so the call counts are auditable, but they total ~112 MB (plausible_distractor alone is 33 MB), which is too much for the repo. The committed evidence stays the summaries + per-case jsonls; the scale_c Also flagging on #237: this corrects that merged capstone too — worth a note there. |
|
Heads up on thread ordering: I expanded my review above rather than adding a second one, so it keeps its original timestamp and does not appear at the bottom where you would look for it. It is now a full sweep of all 24 artifacts with eight requested items: Short version, so you can triage before reading it:
No arithmetic failed anywhere, and the review says at length what is real, which is most of it: the reasoned-versus-bare result, the dissenter break, confidence elasticity, the anchored-seed replication, and above all your clean-control design, which our full draft now cites as the general remedy for this whole family of tautologies. |
… the referee claim sebasmos review found four headline numbers that cannot fail by construction. Verified all four against the committed per-case data and corrected them: - authority "vs control", majority "vs isolated": the baseline arm is identically 0 by construction (the seed is chosen != the holdout's own answer), so a McNemar against it is one-sided. orchestrator "wrong-orch": the leader's forced answer IS the committee output, so it is identically 1. Drop all three degenerate contrasts from the stats_reconciliation family and enter the effects through their non-degenerate rungs (guideline vs colleague, 2-peer vs 1-peer, wrong-peer vs honest-orch). The corrected family is 8 contrasts, 4 survive Holm (was an inflated 7 of 9). Measured properly, the majority effect is null (+0.04, p=1.0) and the orchestrator effect is n.s. (+0.10, p=0.21). - referee "deployable" precision/recall 1.0 is an identity: peers are forced to the shortcut, so the inferred shortcut IS that answer and the decision reduces to the adoption label. Withdraw the "deployable matches the oracle" headline; the only real detector is the LLM judge (recall 0.91). - unanimity summary read said "NOT significant" while p=6.1e-05 and it survives Holm; corrected. RESULTS.md sections 6, 8, 9, 11 rewritten to state these honestly.
…ually measurable The planted arm alone is degenerate (sebasmos, #368): the peers are forced to the shortcut and `wrong` is chosen != the holdout's bare answer, so `deployable`, `oracle` and `adopted` all collapse to board==wrong and every metric is a tautological 1.0. referee_deployable now also runs a clean-cascade control per case -- the same holdout and peers, but the peers assert the CORRECT answer, so no shortcut is planted and adoption is false by construction. A deployable flag there is a real false positive: the referee mistaking honest peer-correction for shortcut adoption, which is the discrimination the planted-only arm cannot test. Re-ran on MedMCQA (40 cases, +80 clean-control calls). Scored over planted + clean, the deployable referee is precision 0.74, recall 1.0, FPR 0.14 -- it catches every adoption but false-alarms on 8 of 57 clean cascades, the real cost of having no key. The oracle (with the key) stays 1.0, so that gap is exactly what the key buys; the LLM judge is recall 0.91 / precision 1.0. RESULTS.md section 11 now reports these instead of the tautological 1.0/1.0, and the summary keeps the planted-only figure labelled DEGENERATE for transparency.
…truction (#389) Five screens over the committed tree, all reading artifacts and ASTs rather than prose. 1. constant_column: a scored binary column with one value across a cohort. 2. hardcoded_verdict: a significance phrase interpolated with a p-value, ast-parsed so docstrings do not trip it. 3. rounded_pvalue: a p stored as exactly 0.0 or 1.0, with exculpations for multiplicity-clamped 1.0 and for a genuine exact 1.0 on a near-even discordant table. 4. duplicate_column: two VARYING binary columns identical or exact complements on every row. This is the shape the first three cannot see, and it is the defect behind #357, #366, #368 and #390. 31 findings, including adopted, deployable and oracle identical on all 40 rows in both text referee lanes, and clean_correct as the exact complement of iso_adopt on 150 rows in three CheXpert arms (#387). 5. forced_direction: a paired test whose losing cell is empty because a saturated rate sits beside it, so its p measures n rather than the effect (#391). Requires the saturation, so a genuine one-sided result such as our own 0-versus-71 is not flagged. PREEXISTING_CEILING caps the exemption map so a new defect cannot be absorbed by adding a line of JSON; verified by planting a defect, exempting it, and watching the suite go red. Anti-vacuity anchors live on synthetic fixtures rather than on the backlog, after @Agastya191 showed the original design punished anyone who fixed a #374 item: he corrected unanimity_break.py properly, deleted its exemption, and the guard own tests went red. Now each screen asserts it fires at all, so fixing a defect keeps the suite green. Verified against his exact scenario. 1117 passed, 7 skipped. 130 findings, 0 unexplained, 1.6s. ruff clean. Reviewed by @Agastya191, whose two catches (the backlog-anchored tests, and the exemption-file escape hatch they created) are both fixed here.
…eree_deployable.py conflict Combines #377's board-render fix (render_board, so a peer's rationale can reach the holdout) with this branch's own clean-cascade control (a second, honest-peer arm so deployable no longer collapses to the adoption label it is scored against). Both touch the same _Hold class; verified render_board's default output (show_rationale=False) is byte-identical to the old inline board join, so the committed MedMCQA cache still replays this exact code path. The #389 degeneracy guard, scanning MedMCQA for the first time, flagged 27 findings. All independently re-verified: 20 rounded p-values are round(p, 6) underflowing genuinely tiny exact McNemar values (checked against benchmaxxing.stats.mcnemar directly, none are computation bugs); naive is constant True because the naive gate cannot separate honest agreement from adoption in either of this branch's cascade arms, exactly the paper's own claim about it; oracle_audit duplicates adopted by the script's own documented design ('upper-bound only'), distinct from the pre-#368 defect where deployable also duplicated it. Three findings (control_adopt, isolated_adopt, wrong_orch_output_wrong, neither_adopt) resolve open human-call items already sitting in the MedQA half of the exemptions file; migrated those out of preexisting into allowlist with the verified reason. Three stale MedQA exemption keys (contrasts.1/3/8) were removed: #368's own capstone regeneration already fixed or dropped those contrasts. All 1117 tests pass.
…rarchy, controls, referee + corrected family (#368) Rebased onto current main and the referee_deployable.py conflict against #377 resolved by hand: combined #377's render_board fix with this branch's own clean-cascade control (verified byte-identical default output, so the committed cache still replays). Items A-G from my change request addressed: board renderer routed through render_board, Holm count corrected to 4/8, oracle/deployable tautology labeled and fixed with a clean-control arm, recall-1.0 forced-ness noted, manifest question deferred (tracked separately). Item H (clean_a's control) filed as #394, not blocking. The #389 degeneracy guard flagged 27 findings on this cohort scanned for the first time; all independently re-verified against benchmaxxing.stats.mcnemar and the underlying scripts, none are computation bugs, all now recorded in tests/degeneracy_exemptions.json with the specific reason. All 1117 tests pass.
…eree, so the cell is measurable The MedQA deployable referee was a dash in the paper: withdrawn as definitional rather than measured. On the planted arm alone the peers are scripted to the shortcut and `wrong` is chosen to differ from the holdout's bare answer, so `deployable`, `oracle` and `adopted` reduce to the same expression and 1.0/1.0/0.0 was forced on all 40 rows. #368 found the remedy and it is a design change, not a relabelling: add an honest-peer arm where both peers assert the CORRECT answer. Shortcut adoption is then false by construction, so a deployable flag there is a genuine false positive, the referee having no key with which to distinguish honest peer correction from being lured onto a wrong answer. MedQA is a lane we own, so this runs it. Result over planted plus clean, 40 cases and 80 rows: naive gate P 0.188 R 1.000 FPR 1.000 tp 15 fp 65 tn 0 deployable P 0.682 R 1.000 FPR 0.108 tp 15 fp 7 tn 58 oracle audit P 1.000 R 1.000 FPR 0.000 tp 15 fp 0 tn 65 The deployable referee costs 7 false positives across 65 honest-peer cases, and that gap from the oracle is the real price of having no answer key. It lines up with MedMCQA's 0.742/1.000/0.140, which is reassuring for both since they were produced independently. The predicate is now genuinely falsifiable here rather than merely non-tautological: `deployable` disagrees with `adopted` on 7 of 80 rows, where before it agreed on 40 of 40. The degeneracy guard confirms this without being told: `duplicate_column` no longer flags `adopted vs deployable` on this file, while it still flags `adopted vs oracle`, which is right, because the oracle's clean-arm verdict is assigned rather than measured. So two exemptions are stale and deleted, and PREEXISTING_CEILING ratchets 122 to 113. That is the guard's ratchet working as designed: fixing a defect forces the baseline down rather than letting it sit. Only artifacts change here. The runner already carries the clean arm on main, since #368's merge brought the shared template in, so this commit is the data plus the ratchet. Verified additive rather than a rewrite in three ways: the manifest was confirmed first by replaying the pre-change script keylessly and regenerating the previously committed summary byte-identically; the planted-arm block in the new summary matches those committed numbers exactly for all three referees; and main's own runner reproduces this entire summary from the committed cache at zero new API calls. 80 new calls, all cached and committed. 1117 passed, 7 skipped. Guard: 151 findings, 0 unexplained. ruff clean. No em dashes.
…eree (#397) Verified: planted-only block matches the pre-PR committed summary exactly (naive 0.375/1.0/1.0, deployable and oracle both 1.0/1.0/0.0, tp15/fp0/tn25 for each). With-clean-control confusion matrix checks out arithmetically (tp15 fp7 tn58 fn0 -> precision 15/22=0.6818, fpr 7/65=0.1077). Deployable no longer duplicates adopted (0/80, was 40/40); oracle still does, by design, matching #368's identical pattern on MedMCQA. Exemptions diff correctly drops the two now-resolved MedQA preexisting entries and deduplicates a few accidentally-doubled keys. All 1117 tests pass on a clean local merge against current main.
3 participants
Summary
Ran the rest of the MedMCQA standard battery on top of the foundational arms in #349: the
committee-structure, seed-shape and control arms, the three that hadn't started (onset #287,
break-it D #290, referee #322), and the battery-wide family correction. All real
gemini-2.5-flash-lite over the medmcqa dev split, same scripts as the MedQA template — no experiment
logic changed. RESULTS.md now covers the whole battery instead of just the four foundational
sections. This closes out #291: every arm (#267–#290 + #322) now has committed results.
Result
contrasts survive Holm. Robust ones — authority guideline +0.92 / senior +0.72 over control,
orchestrator single-point-of-failure (poisons the output 1.0 vs 0.29 for a wrong peer), dissenter
break (unanimous 0.42 → one correct ally 0.03), confidence elasticity +0.26. The two that don't
(true-peer, majority-pressure) sit at the exact-test p=0.125 floor — real in direction but
underpowered, flagged as such.
reasoned ~0.38, p<1e-9). The holdout scrutinizes stated reasoning but defers to bare authority.
transcript plus one private re-query — matches the oracle exactly (precision 1.0, recall 1.0,
catches all 23 adoptions, zero false positives). The naive conformity gate is useless (FPR 1.0).
temp 0 (unanimity makes the metric a tie-break artifact), and the true-peer control passes
(correct peer adopted 0.53 vs wrong 0.20, so the deference isn't blind conformity).
Notes
froze ~3.5h, break-it once). I worked around it with a stall-kill-and-resume watchdog to get these
runs done, and I'm fixing the root cause (a gateway request timeout + retry) in a follow-up.
timeout lands.
experiments/referee/templates that @felipeocampoos's feat: MIMIC-CXR text referee detection, naive gate vs deployable vs judge (#321) #357 ports forMIMIC — I only pointed them at the medmcqa manifest and wrote to the medmcqa results dir, didn't
touch the shared source, so no conflict.
onset_distribution.pyandstats_reconciliation.pygained a--results-dirso they replicate onany dataset; the MedQA defaults are unchanged.