Repository navigation
Releases: criticaldata/benchmaxxing
Release list
Matched control on the MICCAI release
This tag is the frozen state of record for the matched control: the MICCAI 2026 release
(v1.0-miccai2026) plus one commit that adds the request-only baseline, the noise band and the
justification re-read on the same four Gemini cohorts. Nothing else changed.
What the commit adds
experiments/matched_baseline/matched_baseline_reason.py: asks each case again with the same
one-sentence request and no scoring note, same model, temperature 0, replaying the committed baseline
and note responses from their caches, so the note's effect is read net of the request.--repeats
re-asks the control five more times for a noise band.experiments/matched_baseline/justification_reread.py: two model readers score every justification
for a mention of the note, against the keyword screen.- Per-case rows and summaries next to each cohort's committed results as
matched_baseline_reason*;
the credentialed cohort ships as de-identified outcome flags keyed bycase_index. tests/test_matched_baseline_reason.py: every summary regenerates from its rows with no API key.
Cite this tag, not main
main continues past this point with follow-up work that is not part of this control. If you are
checking a number from the matched control, check out v1.1-perspective-control.
No dataset is redistributed here
Each cohort's data carries its own terms and must be obtained from its provider. This repository
contains no report text, images, caches, or subject and study identifiers.
MICCAI 2026 camera-ready: Agents Catching Agents
This tag is the frozen state of record for the paper. Everything the paper claims was produced from
this commit.
Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems
2nd Agentic AI for Medicine Workshop @ MICCAI 2026, paper 0040, poster, Strasbourg, 1 October 2026.
- Paper: arXiv:2608.03744 | Hugging Face
- Licence: MIT
Cite this tag, not main
main continues past this point with follow-up work, including evaluation lanes on vendors other
than Gemini. Those runs write new result artifacts and are not part of the paper. If you are checking
a number from the paper, check out v1.0-miccai2026.
No dataset is redistributed here
MedQA-USMLE, MedMCQA, NIH ChestX-ray14, CheXpert, SUPPORT2 and MIMIC-CXR each carry their own terms
and must be obtained from their own providers. MIMIC-CXR requires PhysioNet credentialing. This
repository contains no MIMIC report text, images, caches, or subject and study identifiers.