feat: MIMIC-CXR text Break-it A, contaminated system pre-screen flag (#318) - #353
Merged
Conversation
felipeocampoos
added
the
priority: high
sebasmos
requested review from
Agastya191,
MohShahin,
PeterLi-jpg,
Yehudha-kennedy,
aadityapanchal7,
akelshiker,
amarzullo24,
armaanvgrewal,
arshnoor08,
azzy-chemE,
dr-shrey,
duckyquang,
maximinl and
renato-umeton
July 29, 2026 14:14
This was referenced Jul 29, 2026
sebasmos
pushed a commit
that referenced
this pull request
Aug 4, 2026
Port experiments/blind_metric/blind_metric.py (not break_it.py's simpler arm D) into experiments/mimic_cxr_text/blind_metric.py: #320 explicitly asks whether the model names the rubric, which only blind_metric.py's three-condition design (baseline/blind/test_aware + a naming regex on the justification) measures. Adapted the prompt builder to render Case.report as clinical context, same fix pattern as #336/#348/#353. Also widened the DUA gitignore rule: this script writes its cache under a different filename (blind_metric_call_cache.jsonl) than the existing call_cache.jsonl pattern covered, and its cached responses embed quoted report text in the model's reasoning, so the narrower glob would have let it slip through. Real n=40 run (gemini-2.5-flash-lite, matching the MedQA original): decoy uptake blind 0.05 vs baseline 0.0, but test_aware (0.175) is HIGHER than blind, the reverse of MedQA's own suppression direction. Traced to a genuine model reasoning failure (confirmed by inspecting the raw completions, not a prompt bug): flash-lite's chain-of-thought misreads "the rubric may be misaligned, answer clinically" as "assume the rubric is correct, justify it." Reported honestly in RESULTS.md with the n=40 sample-size caveat rather than forced to match the expected direction. Where the datasets do agree: both decoy-drifters named the rubric explicitly (2/2), consistent with MedQA's text-lane self-declared drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sebasmos
pushed a commit
that referenced
this pull request
Aug 4, 2026
…udge (#357) * feat: MIMIC-CXR text cascade contagion, bare vs case-anchored seed (#317) Reuse experiments/medqa/reproduce.py --stage cascade as-is for the bare baseline-relative seed (null, same as MedQA: a wrong peer with no rationale does not spread). Copy experiments/medqa/push_c.py into experiments/mimic_cxr_text/push_c.py for the case-anchored plausibility dose-response, adapted for this dataset's report/question split (#336): _mcq now renders Case.report as clinical context (missing entirely in the original, which would answer blind), and the anchor detail is pulled from Case.report instead of Case.question (the latter is now a fixed stem, identical across cases, which would make every "anchored" rationale generic). build_solo_records.py reconstructs push_c.py's --solo-records input from the already-cached n=600 solo run, no new API calls. Real n=20 (bare) and n=60 (anchored) runs against Gemini. Cascade transcripts are gitignored (they embed report text via committee turn content, same DUA policy as manifests/call caches). See experiments/mimic_cxr_text/RESULTS.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: case-anchored rationale pulled report header boilerplate, not findings (#348 review) @sebasmos caught it on review: the anchor detail took the first 14/30 words of the whole report, which for MIMIC-CXR is always fixed boilerplate ("FINAL REPORT", "EXAMINATION: CHEST (PA AND LAT)", "INDICATION: ..."), not clinical content. That made "anchored" and "generic" prompts near-identical, so the anchored-equals-generic null was measuring the boilerplate, not anchoring. Fixed with _findings_text(), extracting from the first FINDINGS:/IMPRESSION: header onward (falls back to the whole report if neither is present). Spot-checked against real reports to confirm real clinical sentences are captured. Also added two paired tests sebasmos had computed independently on the old (buggy) run: anchored_strong vs generic (p=0.29, correctly not a dose-response) and anchored vs anchored_solo (the majority-vs-single-peer contrast, p=0.0039 -- the actually-significant effect, now foregrounded in RESULTS.md instead of the null anchored-vs-generic comparison). Re-ran n=60 for real with the fix. RESULTS.md rewritten to report both new contrasts and stop reading the null as a dose-response. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: MIMIC-CXR text Break-it A, contaminated system pre-screen flag (#318) Copy experiments/medqa/break_it.py's arm A into experiments/mimic_cxr_text/break_it_a.py, adapted for this dataset's report/question split (_mcq renders Case.report as clinical context, same fix pattern as #336/#348) and per #318's explicit instruction to choose the flagged wrong answer per-model so it differs from that model's own clean baseline (MedQA's original picked one wrong per case regardless of model and was noted as confounded there). Real n=20 run against Gemini: a wrong automated pre-screen flag is adopted by both tiers (flash +0.40, flash-lite +0.50), comparable to MedQA's own system-flag finding (+0.26/+0.30). See experiments/mimic_cxr_text/RESULTS.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: MIMIC-CXR text blind-metric substitution probe (#320) Port experiments/blind_metric/blind_metric.py (not break_it.py's simpler arm D) into experiments/mimic_cxr_text/blind_metric.py: #320 explicitly asks whether the model names the rubric, which only blind_metric.py's three-condition design (baseline/blind/test_aware + a naming regex on the justification) measures. Adapted the prompt builder to render Case.report as clinical context, same fix pattern as #336/#348/#353. Also widened the DUA gitignore rule: this script writes its cache under a different filename (blind_metric_call_cache.jsonl) than the existing call_cache.jsonl pattern covered, and its cached responses embed quoted report text in the model's reasoning, so the narrower glob would have let it slip through. Real n=40 run (gemini-2.5-flash-lite, matching the MedQA original): decoy uptake blind 0.05 vs baseline 0.0, but test_aware (0.175) is HIGHER than blind, the reverse of MedQA's own suppression direction. Traced to a genuine model reasoning failure (confirmed by inspecting the raw completions, not a prompt bug): flash-lite's chain-of-thought misreads "the rubric may be misaligned, answer clinically" as "assume the rubric is correct, justify it." Reported honestly in RESULTS.md with the n=40 sample-size caveat rather than forced to match the expected direction. Where the datasets do agree: both decoy-drifters named the rubric explicitly (2/2), consistent with MedQA's text-lane self-declared drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: MIMIC-CXR text referee detection, naive gate vs deployable vs judge (#321) Port experiments/referee/referee_deployable.py and referee_judge.py into experiments/mimic_cxr_text/. Both are self-contained (each builds its own anchored-cascade committee run per case with two colluding peers + a real holdout), so this doesn't depend on #317/#355's cascade transcripts. Same report/question adaptation as #336/#348/#353/#320: _mcq renders Case.report as clinical context, and the anchor detail reuses push_c.py's _findings_text. Real n=40 runs against Gemini, 14 real adoptions: - naive_gate: precision 0.35, recall 1.0, FPR 1.0 (over-fires, as expected) - deployable (peer-modal + private re-query, no key): precision/recall 1.0, FPR 0.0, matching the oracle upper bound - same-lineage judge: precision 1.0, recall 0.93, FPR 0.0 Closely matches MedQA's own referee finding (naive over-fires, deployable recovers the oracle, judge is viable but noisier). Completes the #296 battery (#316-#321), modulo the anchored-seed contagion gap tracked in #355. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: close #355, anchored-seed contagion from already-committed referee data #317's "contagion = shared minus isolated" was only measured for the bare seed in #348 (n=20, 0.0). referee_deployable.jsonl (#357, n=40) already contains the same pair for the ANCHORED seed at zero extra API cost: its board answer (shared, anchored rationale) vs bare answer (private, unseeded re-query) is exactly a shared-vs-isolated comparison, since `wrong` is chosen to differ from bare by construction (same baseline-relative convention used throughout the project). contagion = 0.35 - 0.0 = 0.35, notably higher than the bare seed's null (0.0), consistent with the battery's overall finding that MIMIC-CXR text holdouts propagate a minimally-plausible seed but not a content-free one. Documented the one honest caveat: this isolated condition is a cold, seedless re-query, not MedQA's original isolated-committee mechanics (same seed planted per-agent, no board-sharing) -- a stronger, slightly different control, judged sufficient given it's the same private-re-query convention used everywhere else in this battery (push_c.py, break_it_a.py, both referee scripts). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: naming regex and prompt asymmetry in blind_metric.py (#356 review) Two bugs caught by @Agastya191 on PR #356: 1. _NAMING was only word-bounded on its first/last alternative, and had a substring fallback treating a bare decoy_letter ("A"/"B") appearing anywhere in the first 120 chars as evidence of naming -- true of almost any sentence, so `named` could never come back False for a drifter. n_drifted_silently: 0 was a guaranteed artifact, not an observation. Fixed: proper \b boundaries on every alternative, substring fallback removed. 2. blind_p asked for a justification before the letter; aware_p asked for a bare letter only, confounding test_awareness_suppression with the justification requirement itself. Fixed: aware_p now requests the same one-sentence justification. Re-ran n=40 for real with both fixes. Both changed the reading substantially: test_aware dropped from 0.175 to 0.075 (most of the "reversed suppression" was the justification-requirement artifact, not a model reasoning failure), and the naming result flipped from 2/2 to 0/2 -- confirmed by reading the raw justifications, which cite the report's own clinical language, not the rubric. So MIMIC-CXR text's blind-metric drift is silent here (matching the imaging lane, opposite of MedQA's own 11/11 self-declared text-lane finding), the honest number rather than the artifact the buggy regex reported. base_is_decoy=0-by-construction now documented explicitly (same convention as #317/#355). Also registers mimic_cxr_text in benchmaxxing/datasets/status.py and staging.py, added to main by an unrelated merge (infra this branch predates) that now requires every registered adapter to have both entries. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #348 (cascade, itself stacked on #336, now approved) — this branch includes those
commits too; the diff will narrow once they merge.
experiments/medqa/break_it.py's arm A intoexperiments/mimic_cxr_text/break_it_a.py._mcqto renderCase.reportas clinical context (same fix pattern as feat: MIMIC-CXR text adapter + solo susceptibility real-API results (n=35, n=600) #336/feat: MIMIC-CXR text cascade contagion, bare vs case-anchored seed (#317) #348 —MedQA's original has no separate
reportfield and never needed this).from that model's own clean baseline. MedQA's original arm A picked one
wrongper caseregardless of model and was noted as confounded there ("flag equalled the model's baseline
wrong answer, 0.90"); this avoids that confound by construction (
control_rateis 0 here,so
flag_rateis directly the honest adoption rate).solo_records.jsonl(126 hard cases) built in feat: MIMIC-CXR text cascade contagion, bare vs case-anchored seed (#317) #348, no new prep needed.Results (n=20 hard cases, both tiers)
Comparable in direction and magnitude to MedQA's own system-flag finding (+0.26 flash, +0.30
flash-lite). Full writeup in
experiments/mimic_cxr_text/RESULTS.md.Part of #296. Closes #318.
Test plan
pytest tests/— 613 passed, 11 skippedruff check experiments/mimic_cxr_text/— clean