Skip to content

Releases: criticaldata/benchmaxxing

Matched control on the MICCAI release

Choose a tag to compare

@sebasmos sebasmos released this 18 Oct 19:14

This tag is the frozen state of record for the matched control: the MICCAI 2026 release
(v1.0-miccai2026) plus one commit that adds the request-only baseline, the noise band and the
justification re-read on the same four Gemini cohorts. Nothing else changed.

What the commit adds

  • experiments/matched_baseline/matched_baseline_reason.py: asks each case again with the same
    one-sentence request and no scoring note, same model, temperature 0, replaying the committed baseline
    and note responses from their caches, so the note's effect is read net of the request. --repeats
    re-asks the control five more times for a noise band.
  • experiments/matched_baseline/justification_reread.py: two model readers score every justification
    for a mention of the note, against the keyword screen.
  • Per-case rows and summaries next to each cohort's committed results as matched_baseline_reason*;
    the credentialed cohort ships as de-identified outcome flags keyed by case_index.
  • tests/test_matched_baseline_reason.py: every summary regenerates from its rows with no API key.

Cite this tag, not main

main continues past this point with follow-up work that is not part of this control. If you are
checking a number from the matched control, check out v1.1-perspective-control.

No dataset is redistributed here

Each cohort's data carries its own terms and must be obtained from its provider. This repository
contains no report text, images, caches, or subject and study identifiers.

MICCAI 2026 camera-ready: Agents Catching Agents

Choose a tag to compare

@sebasmos sebasmos released this 25 Aug 20:07

This tag is the frozen state of record for the paper. Everything the paper claims was produced from
this commit.

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems
2nd Agentic AI for Medicine Workshop @ MICCAI 2026, paper 0040, poster, Strasbourg, 1 October 2026.

Cite this tag, not main

main continues past this point with follow-up work, including evaluation lanes on vendors other
than Gemini. Those runs write new result artifacts and are not part of the paper. If you are checking
a number from the paper, check out v1.0-miccai2026.

No dataset is redistributed here

MedQA-USMLE, MedMCQA, NIH ChestX-ray14, CheXpert, SUPPORT2 and MIMIC-CXR each carry their own terms
and must be obtained from their own providers. MIMIC-CXR requires PhysioNet credentialing. This
repository contains no MIMIC report text, images, caches, or subject and study identifiers.