Training-data contamination audit for MedQA (and mitigation plan)

[sebasmos]

Problem

MedQA-USMLE is a widely-published benchmark, so parts of it are plausibly in the pretraining data of the models we evaluate. For our design this is mostly controlled (the twin-pair contrast holds the case fixed and only flips a spurious cue, so memorized answers cancel in the paired comparison), but we should audit and document it rather than assume it, because a reviewer will ask, and because heavy memorization could dampen the very flips we measure.

What to build

• A contamination probe: for a sample of MedQA items, measure clean-condition accuracy and a memorization signal (e.g. verbatim-continuation or high-confidence-without-reasoning rate) per model, and report it.
• An argument + evidence that the paired design is robust to memorization: because ground truth is never altered and the cue is answer-preserving, a flip between twins reflects cue reliance, not recall. Quantify this by checking that flip rates are not simply a function of clean-condition accuracy.
• A mitigation option: a held-out or perturbed-stem subset for a sensitivity check, so we can show the effect survives when memorization is less likely.

Acceptance

• A short report (notebook or summary.md section) with per-model clean accuracy, a memorization proxy, and flip rate, plus the paired-design robustness argument.
• A documented sensitivity subset and a rerun of the pilot on it.
• Findings folded into the pre-registration / limitations (Stage 0 - Pilot run and pre-registration (freeze the analysis plan before the main runs) #9, Run the stage-0 pilot on real MedQA and freeze the pre-registration #93).

This is a text-lane (MedQA) concern; the imaging lanes have their own leakage considerations tracked separately. Gemini is the default backend for now.