Goal (apples-to-apples with MedQA/NIH): Single-model shortcut baseline: does an answer-preserving text cue flip the model's answer more often than its own clean-read noise floor?
Copy this template: experiments/medqa/reproduce.py (solo + noise-floor pass). It already implements the experiment; you mostly swap in the MIMIC-CXR text manifest and adapt the loader if needed. Do NOT rewrite the method, replicate it.
What to do: Point it at the MIMIC-CXR text manifest, run flash and flash-lite through the Gemini API. Report per-cue flip rate and the honest quantity, flip-above-noise (noise floor = re-ask the clean item, count changes).
Golden rule: real data + a real Gemini API run only. A passing mock / synthetic smoke test does NOT count. Commit results under experiments/mimic_cxr_text/results/ in the SAME JSON/JSONL format as the source, so the cross-dataset comparison table can be filled directly.
Definition of done: the real keyed-run numbers are committed under results/, and the row/cell for MIMIC-CXR text in the cross-dataset table can be read off them. Part of #296. Uses MIMIC report text; shares loader #45 with the imaging half, coordinate with @amarzullo24.
Goal (apples-to-apples with MedQA/NIH): Single-model shortcut baseline: does an answer-preserving text cue flip the model's answer more often than its own clean-read noise floor?
Copy this template:
experiments/medqa/reproduce.py(solo + noise-floor pass). It already implements the experiment; you mostly swap in the MIMIC-CXR text manifest and adapt the loader if needed. Do NOT rewrite the method, replicate it.What to do: Point it at the MIMIC-CXR text manifest, run flash and flash-lite through the Gemini API. Report per-cue flip rate and the honest quantity, flip-above-noise (noise floor = re-ask the clean item, count changes).
Golden rule: real data + a real Gemini API run only. A passing mock / synthetic smoke test does NOT count. Commit results under
experiments/mimic_cxr_text/results/in the SAME JSON/JSONL format as the source, so the cross-dataset comparison table can be filled directly.Definition of done: the real keyed-run numbers are committed under results/, and the row/cell for MIMIC-CXR text in the cross-dataset table can be read off them. Part of #296. Uses MIMIC report text; shares loader #45 with the imaging half, coordinate with @amarzullo24.