Skip to content

MIMIC-CXR: deterministic nested subset selection + reproducible download #324

Description

@amarzullo24

The full MIMIC-CXR-JPG release is ~600 GB, but the imaging battery only needs a few hundred studies per arm, and the arm sizes nest (blind 100 subset of cascade 150 subset of referee 300 subset of solo/cross-modal 600, with 35 nested inside the 600 for the NIH-matched comparison). This adds tooling to select that nested subset deterministically and download only the images it references.

  • experiments/mimic_cxr_image/build_subset.py: select (rank studies via budget.subsample_cases, write per-arm manifests + download_list.txt) and verify (check images resolve, write sha256 provenance).
  • Reproducibility = deterministic list (from seed) + pinned per-arm manifests + provenance checksums + a standard-tool wget one-liner. No bespoke fetcher, no credentials in the repo, mirroring how the NIH lane pins its data.
  • All images per study; finding-positive studies by default. Manifest CSVs are gitignored (they embed credentialed report text; regenerate from the seed).

Part of #295. Unblocks #92.

Metadata

Metadata

Assignees

Labels

datasetDataset adapter / data stagingdataset:mimic-cxrexperimentExperiment runner / study designpriority: highDo this soon; unblocks the paper or other work

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions