MIMIC-CXR: deterministic subset tooling + imaging battery (closes #324)

[amarzullo24]

Rebased onto main now that #263 has merged: the diff is only the MIMIC-CXR tooling (13 files, no budget changes).

What this adds

Tooling to run the imaging standard battery on MIMIC-CXR without pulling the ~600 GB release:

• experiments/mimic_cxr_image/build_subset.py — select ranks studies by a stable hash of the seed (nested arms: 100 < 150 < 300 < 600, with 35 nested for the NIH match), writes per-arm manifests + download_list.txt; verify checks images resolve and writes sha256 provenance.
• README.md — full reproduction recipe (deps -> select -> download -> verify -> all six arms), with a per-arm --out so the two watermark cascades cannot collide.
• Aggregate *_summary.json for the six arms (seed 20240724, gemini-2.5-flash). Per-case transcripts, caches, manifests, download list, and provenance are gitignored (credentialed report text; regenerate from the seed).
• tests/test_mimic_subset.py — study-level nesting, all-images-per-study, order-invariance, eligibility, URL formatting.

Results

Posted to #295. Committee/cascade effects are strong and consistent; solo susceptibility is at the noise floor on MIMIC (matched 35-vs-35: NIH watermark +0.11 vs MIMIC -0.04).

Note (#332): the cascade contagion cell is being refined by the plant-direction reanalysis. On MIMIC ~80% of cases plant the truth (the model reads pneumothorax poorly), so the honest genuinely-false contagion is ~+0.49-0.51, not the ALL +0.82. A follow-up commit will report the genuinely-false subgroup.

Closes #324. Part of #295. Unblocks #92.

[sebasmos]

This is unblocked now: #263 is merged (real budgeted run passed, and the subsample_cases prefix fix your nested subset relies on is in). Please rebase onto main to drop the now-redundant budget commits, then mark it ready for review and I'll go through the MIMIC-CXR battery. Thanks!

[amarzullo24]

Follow-up per #332/#339: added experiments/mimic_cxr_image/plant_direction_check.py + results/plant_direction_summary.json (offline, no API). Corrected genuinely-false contagion for the cascade-family arms:

• cascade_150 (Cascade contagion: confident wrong read on a cued image (MIMIC-CXR image) #311): +0.49 (ALL +0.82 was inflated by 80% truth-planting)
• referee_300 cascade: +0.51
• strength_cascade (Cue strength / natural-vs-injected cue sweep (MIMIC-CXR image) #313): ~+0.53-0.55, flat across opacity
• system_flag (System-flag vs peer contagion (MIMIC-CXR image) #312): robust, +0.225 ~ ALL +0.200

No re-run needed: imaging_cascade.py is fixed on main (#333) and its genuinely-false subgroup is the honest number; the split reads the committed transcripts. Qualitative conclusions unchanged.

[Agastya191]

good work on build_subset.py and all other files. The study-level nesting, the single download_list.txt for the 600-study superset, the per-arm --out note that keeps the two watermark cascades from colliding, and test_mimic_subset.py covering order-invariance are all solid, and folding in plant_direction_check.py for #332 was the right call.

One problem is that in build_subset.py DEFAULT_SIZES["nih_match"] = 35 counts studies, while select_arms expands every picked study to all of its images. This results in nih_match_35.csv shipping 50 images rather than 35 (selection_summary.json records studies: 35, images: 50, and results/nih_match_35/imaging_solo_summary.json reports n: 50), so the matched comparison headlined in the description is 50-vs-35 against NIH's n: 35, and those 50 images cluster into 35 studies, meaning the MIMIC flip rate averages over correlated repeat views of the same study while the NIH set is one image per case.

You have multiple options to fix this. For example, I would size the nih_match arm at the image level, walking studies in rank order until the arm hits exactly 35 images, so the matched arm is genuinely the same n and the same one-image-per-observation shape as the NIH run.

Requesting changes only because that arm's number is going into the cross-dataset table on #295; everything else here reads ready to merge. Separately, and not worth blocking on: section 4 of the README doesn't include the imaging_solo --manifest $M/nih_match_35.csv invocation that produced the committed results/nih_match_35/ summary, so the recipe doesn't currently regenerate all six arms.

[sebasmos]

This is great work, exactly what was missing for the cross-dataset table. One thing before merge: nih_match is sizing by study and ending up with 50 images instead of 35, so let's walk it study by study until it hits exactly 35 images, then this is ready to go.

[maximinl]

Peer review — agree with Agastya and Seb on the nih_match sizing; everything else looks merge-ready.

Solid

• Study-level nesting via subsample_cases, single download_list.txt for the 600-study superset, verify sha256 provenance, and order-invariance tests are the right reproducibility shape for credentialed MIMIC without pulling 600 GB.
• plant_direction_check.py folding in the #332 correction (ALL contagion inflated by truth-planting; genuinely-false ~+0.49–0.53) is exactly what the cascade cell needs before the cross-dataset table cites it.
• DUA handling (gitignore caches/manifests, commit aggregates) matches the lane.

Blocker: nih_match is 35 studies → 50 images
Confirmed in the committed artifacts: selection_summary.json has nih_match: {studies: 35, images: 50} and results/nih_match_35/imaging_solo_summary.json reports n: 50. The matched NIH comparison is therefore 50-vs-35, and the 50 MIMIC rows are correlated multi-view clusters while NIH is one image per case.

Please size nih_match at the image level: walk studies in rank order until the arm hits exactly 35 images (as Agastya suggested), regenerate that arm’s solo summary, and update the #295 table to the corrected n. Not blocking on the README gap Agastya noted (missing imaging_solo --manifest …/nih_match_35.csv in §4), but worth fixing in the same pass so the recipe regenerates all six arms.

[sebasmos]

The five non-matched arms are real and the provenance work (seeded manifest, sha256 per image, download list) is solid.

Still the same single blocker on head: nih_match is 35 studies but 50 images, so the matched cell is 50-vs-35 against NIH's 35 and is not comparable. The fix is small: walk studies until the image count hits 35, update the nesting test, rerun that one arm.

No rush given you are travelling, and if it is easier, someone with PhysioNet access can push that one fix rather than it waiting on your connection. Everything else here I am happy with.

[amarzullo24]

Fixed the nih_match sizing blocker in d3331d0 -- thanks @Agastya191 @sebasmos @maximinl for the unanimous catch.

• nih_match is now sized by image count: the first 35 images in study-rank order (IMAGE_SIZED_ARMS in build_subset.py), so it's genuinely n=35 one-image-per-observation rather than 35 studies expanded to ~50 correlated views. selection_summary.json now records nih_match: {studies: 26, images: 35}.
• Verified on the real manifests that it still nests: nih ⊂ blind ⊂ cascade ⊂ referee ⊂ solo.
• Split the study-level nesting test from a dedicated image-sizing + determinism test.
• README §4 now includes the nih_match_35 imaging_solo invocation (Agastya's note), so the recipe regenerates all six arms.
• Removed the stale n=50 solo summary; the n=35 one regenerates in the real-data rerun (it's a cached subset of solo_600 plus the noise floor), and I'll update the MIMIC-CXR imaging standard-battery replication (tracking) #295 matched cell to n=35 once that lands.

Left Closes #343 to @Agastya191's #347 per @sebasmos's note (drop it until the real rerun lands).

[sebasmos]

Image-count sizing is the right call and the nesting still holds, so the matched cell is finally comparable to NIH's 35. Splitting the determinism test out from the nesting test is a good touch. Approving. Note the matched-cell number in the paper stays pending until the real rerun lands, which is #347's ask.

Updated 2026-07-29: merging this is now the first step in unblocking that rerun, so it is worth doing ahead of the other imaging PRs. #347 is based on this branch and cannot run two of its six arms until it sits on main, because imaging_system_flag.py and imaging_strength_cascade.py here still carry the pre-#338 plant direction and #347's own guard correctly refuses them. Once this lands, #347 rebases onto main, picks up the fixed runners, and the full battery can run.

One process note so this does not stall: @Agastya191 and @maximinl both still have change requests on this PR from before d3331d0, so it cannot merge on my approval alone. A re-review from either of you, or dismissing the stale reviews, is the only thing left.