feat: MIMIC-CXR text adapter + solo susceptibility real-API results (n=35, n=600)

[felipeocampoos]

Summary

• Adds benchmaxxing/datasets/mimic_cxr_text.py (Lane B): builds a 4-way MCQ per study from MIMIC-CXR report text + CheXpert labels, per the design agreed in Design: MCQ template + distractor selection for mimic_cxr_text adapter (blocks #316) #330 (exactly one confirmed-positive finding as the answer, 3 confirmed-negative findings as distractors, "No Finding" studies excluded, uncertain labels never used as distractors).
• Registers the adapter and updates the dataset registry/smoke tests.
• Reuses experiments/medqa/reproduce.py unchanged (per Solo shortcut susceptibility + noise floor (MIMIC-CXR text) #316's "do not rewrite the method, replicate it") to run real Gemini API solo-susceptibility + noise-floor experiments on the new manifest, at n=35 (smoke) then n=600, per Real-API solo susceptibility run: MIMIC-CXR text at n=35 then n=600 #334.
• Commits only aggregate results (solo_results.json, RESULTS.md); manifests and call caches are gitignored since MIMIC-CXR report text is credentialed PhysioNet data under a DUA and must not be redistributed, even in this private repo.

Results (n=600)

model	overall flip	noise floor	flip-above-noise
gemini-2.5-flash	0.079	0.000	+0.079
gemini-2.5-flash-lite	0.124	0.000	+0.124

Consistent with the n=35 smoke run (0.095 / 0.124). Full detail in experiments/mimic_cxr_text/RESULTS.md.

Closes #316. Part of #296. Related: #330, #334.

Test plan

• pytest tests/ — 611 passed, 10 skipped (full suite, python 3.12)
• ruff check clean on all changed files
• Smoke-tested build_manifest against the real credentialed MIMIC-CXR data at /orcd/pool/006/lceli_shared/data/mimic-cxr-{reports,labels}
• Real Gemini API run at n=35 and n=600 (results committed)

@sebasmos — flagging for review given the #330 design decisions and the credentialed-data handling call (gitignoring manifest/call_cache); let me know if that split should be different.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[armaanvgrewal]

@felipeocampoos @sebasmos

Tested locally on macOS Apple Silicon in a .venv. ruff check . failed with 6 lint errors, all in assets/figures/figure1_gen.py.

Environment/details:

python --version: Python 3.14.6
ruff --version: ruff 0.15.22
branch: pr-336-mimic-cxr-text
git status: clean working tree
install used: pip install -e ".[dev]"

Commands run:

pip install -e ".[dev]"
ruff check .
pytest -q

Results:

pip install -e ".[dev]"  # completed successfully
pytest -q                # 680 passed, 6 skipped

ruff check . failed with 6 lint errors, all in assets/figures/figure1_gen.py:

assets/figures/figure1_gen.py:36:3: F541 f-string without any placeholders
assets/figures/figure1_gen.py:65:14: E702 Multiple statements on one line (semicolon)
assets/figures/figure1_gen.py:139:3: F541 f-string without any placeholders
assets/figures/figure1_gen.py:140:3: F541 f-string without any placeholders
assets/figures/figure1_gen.py:150:27: E702 Multiple statements on one line (semicolon)
assets/figures/figure1_gen.py:151:27: E702 Multiple statements on one line (semicolon)

So the test suite passes locally, but the PR does not currently pass ruff check . on my setup.

[felipeocampoos]

@armaanvgrewal thanks for testing! Confirmed this is pre-existing on `main`, not something this PR introduces: our branch has zero commits/diff touching `assets/figures/figure1_gen.py`, and `ruff check` fails identically against a clean checkout of `main` (tried in an isolated worktree, same file, same errors — f-strings without placeholders + a missing-executable-bit shebang warning). Likely a ruff version/config difference between your local setup and whatever ran clean before (some of these rules, like EXE001, are newer). Not blocking this PR — happy to fix it here anyway if useful, or it's better as its own small cleanup PR against main. Let me know which you'd prefer.

[felipeocampoos]

Fixed in #341 (small, separate PR against main since it's unrelated to this diff). Once that merges, a rebase here will pick it up.

[Agastya191]

Good work on mimic_cxr_text.py and on the #330 filtering — keeping only single-confirmed-positive studies and drawing distractors solely from confirmed negatives is the right call, and the DUA split holds up: I read run_solo and solo_results.json plus call_cache.jsonl really is everything it writes, both of which your .gitignore covers. The problem is that build_manifest puts the whole report into question instead of report, and lexical_overlap_bias in cues/text.py appends every salient question token to a distractor. I can't run it against the credentialed text, but on a typical-length report that is ~67 tokens, so the distractor becomes a ~540-character blob that names the correct finding verbatim and is also ~45x longer than every other option — which means the lexical_overlap rate that RESULTS.md calls the strongest cue for both tiers (0.093 and 0.148) is partly counting models that read the answer out of the report pasted into option A, while longest_option fires on the same case.

You have a few options here. I would put the report in Case.report and leave question as the short stem, which is what pubmedqa.py and mimic_cxr.py already do — _mcq_prompt renders report as "Clinical context:", so the model sees identical text, but the cue then draws from the stem alone (I checked: the distractor drops to Cardiomegaly primary finding described report). That does mean re-running the n=600 solo pass.

[sebasmos]

Nice work getting this running end to end. Agastya's catch is a good one though: report should go in Case.report, not question, since the cue is pulling the whole report into a distractor right now. That's probably why lexical_overlap looks strongest. Worth fixing and re-running the n=600 pass before we trust that number.

[maximinl]

Peer review — agree with Agastya and Seb; requesting the schema fix + n=600 re-run.

What’s solid

• #330 filtering (single confirmed-positive answer, distractors from confirmed negatives only, drop No Finding / multi-positive / underdetermined) looks correct, and the tests pin those exclusions well.
• DUA split is right: gitignore manifests + call caches, commit only aggregates.
• Reusing experiments/medqa/reproduce.py unchanged matches #316’s replicate-don’t-rewrite rule.
• Zero noise floor on both tiers is a clean control signal.

Blocker: report text in Case.question
build_manifest does:

question=f"{report_path.read_text(...)}\n\n{_QUESTION_PROMPT}"

lexical_overlap_bias in cues/text.py appends salient question tokens onto a distractor. Putting the full report in question therefore (a) pastes the correct finding into the contaminated option and (b) makes that option ~orders of magnitude longer, so longest_option co-fires. That is a plausible explanation for why RESULTS.md ranks lexical_overlap as the strongest cue (0.093 / 0.148) — part of that rate may be models reading the answer out of the option, not cue susceptibility.

Put the report in Case.report and leave question as the short stem (same pattern as pubmedqa / mimic_cxr). _mcq_prompt still surfaces the report as clinical context, so the model sees the same text; the cue only draws from the stem. Then re-run the n=600 solo pass before those numbers are citable.

(Armaan’s ruff note on figure1_gen.py is orthogonal and already fixed in #341.)

[felipeocampoos]

@Agastya191 @sebasmos @maximinl — fixed, and found a second bug along the way. Pushed in 1907034.

Bug 1 (your catch): moved report into Case.report, question is now just the fixed short stem, matching pubmedqa.py. Confirmed with a direct build_text_twin spot-check that the lexical_overlap distractor no longer quotes report text.

Bug 2 (found while re-running): re-running with just bug 1 fixed produced a wildly different, much higher flip rate (~0.6-0.9) and a ~60% abstention rate on gemini-2.5-flash (lots of "please provide the report" replies). Traced it to benchmaxxing/data.py's _text_case, which never reads report back off a manifest CSV (unlike _image_case, which does) — a pre-existing bug in shared code, not in this adapter. Since reproduce.py's run_solo loads cases via load_cases(), every real API call was silently missing the report entirely; the model was guessing cold among 4 similar finding names. Fixed with a one-line read-back + a regression test (test_text_case_report_round_trips).

With both fixes, re-ran n=35 and n=600 for real:

{"gemini-2.5-flash": {"flip": 0.089, "noise_floor": 0.067},
 "gemini-2.5-flash-lite": {"flip": 0.114, "noise_floor": 0.0}}

Also added refusal_aware_reanalysis.py — re-derives flip rate from the already-cached responses using benchmaxxing.extract's proper abstention-aware parser (no new API calls), as a cross-check that the numbers aren't still contaminated by unhandled refusals: abstention rate is ~0% at both n=35 and n=600, and it matches reproduce.py's own numbers closely.

Full writeup in experiments/mimic_cxr_text/RESULTS.md. Ready for re-review.

[Agastya191]

Good work on the fix, and on running down the data.py bug. I checked it: _text_case really never read report back while _case_to_row had been writing the column all along, and no committed manifest under experiments/ has a populated report column, so the one-line read-back can't move any existing MedQA number. One problem, though: now that question is the constant _QUESTION_PROMPT, lexical_overlap_bias in cues/text.py has no case-specific text left to draw from, since _content_tokens on that stem returns ['primary', 'finding', 'described', 'report'] for every study, so the cue appends the identical four-word suffix to option 0 on all 600 cases. That makes lexical_overlap the same manipulation as longest_option with a different filler, rather than the per-case lexical signal it is on medqa.py and pubmedqa.py where question carries the case, so the two columns converging isn't a finding and RESULTS.md's cross-dataset read of that column isn't comparing the same construct.

You have a few options. I would have lexical_overlap_bias fall back to case.report tokens minus any token appearing in the options when the stem yields nothing case-specific, which restores a per-case signal without putting the report's naming of the answer back into the distractor. If you'd rather not touch shared cue code on this PR, a sentence in RESULTS.md saying the cue degenerates to a constant suffix here would be enough to stop that number being read as comparable to MedQA's. Separately, that same paragraph calls lexical_overlap strongest for both tiers, but results_n600/solo_results.json has flash-lite at longest_option 0.130 against lexical_overlap 0.118.

[sebasmos]

Bug 2 is genuinely fixed, I round-tripped a manifest and confirmed the report renders as Clinical context in the built prompt. Good catch finding it while re-running rather than shipping the first fix.

One new problem though. Now that the report is out of question, lexical_overlap_bias draws its tokens from the fixed 8-word stem, so it appends the same constant suffix to the same distractor that longest_option pads, on all 600 cases. Those two columns are one manipulation, not two, so the per-cue split is not comparable to MedQA/PubMedQA where the overlap is case-specific.

Also RESULTS.md says lexical_overlap is the strongest cue for both tiers at n=600, but the committed results_n600/solo_results.json has flash-lite longest_option 0.13 above lexical_overlap 0.1183.

Flagging that I have provisionally put your n=600 cross-dataset row in the paper with that cell marked on hold, so resolving this unblocks it.

[felipeocampoos]

@Agastya191 @sebasmos — good catch, and confirmed exactly as described: with question now a
constant stem, lexical_overlap_bias was appending the same 4-word suffix on every case,
collapsing it into longest_option with different filler. Fixed in benchmaxxing/cues/text.py
(1907034 → 34c20a7):

• The cue now draws its perturbation tokens from case.report when present, falling back to
  case.question unchanged for datasets with no separate report field (MedQA etc. — confirmed
  unaffected, case.report is None there so this is a no-op for them).
• Also excludes any token appearing in any option (not just the target), not only the ones
  already in the code — so the injected text can't echo another option's wording, most
  importantly the correct answer, back into a distractor.
• Two regression tests added (test_lexical_overlap_draws_from_report_not_a_fixed_question_stem,
  test_lexical_overlap_never_injects_another_options_wording) covering exactly this failure
  mode so it can't regress silently again.

Re-ran n=35 and n=600 for real. lexical_overlap and longest_option now diverge per model as
expected of two genuinely different manipulations:

{"gemini-2.5-flash": {"lexical_overlap": 0.093, "longest_option": 0.105, "option_order": 0.043},
 "gemini-2.5-flash-lite": {"lexical_overlap": 0.175, "longest_option": 0.13, "option_order": 0.095}}

@Agastya191 also right that RESULTS.md's "lexical_overlap strongest for both tiers" line was
wrong even on its own (pre-fix) numbers — flash-lite already had longest_option 0.130 >
lexical_overlap 0.118. Corrected: no single cue is strongest for both tiers now, and RESULTS.md
says so explicitly instead of overclaiming. Full writeup updated in
experiments/mimic_cxr_text/RESULTS.md.

Ready for re-review.

[sebasmos]

Verified the post-fix n=600 numbers on the branch: flash 0.043 / 0.105 / 0.093 and flash-lite 0.095 / 0.130 / 0.175, so the two cues genuinely diverge now. Drawing from the report and excluding tokens present in any option is the right fix, and the two regression tests pin exactly the failure mode. Dropping the overclaim rather than defending it is appreciated. Approving.