Skip to content

Referee detection: naive gate vs targeted vs deployable (MIMIC-CXR image) #315

Description

@sebasmos

Goal (apples-to-apples with MedQA/NIH): Referee (the named contribution): naive conformity gate vs targeted cross-condition referee vs deployable monitor (transcript + one private re-query) on the {ds} imaging cascade. Report precision / recall / FPR.

Copy this template: experiments/imaging/imaging_referee.py and imaging_judge_referee.py. It already implements the experiment; you mostly swap in the MIMIC-CXR image manifest and adapt the loader if needed. Do NOT rewrite the method, replicate it.

What to do: Run on the MIMIC-CXR image cascade transcripts; report P/R/FPR for each referee.

Golden rule: real data + a real Gemini API run only. A passing mock / synthetic smoke test does NOT count. Commit results under experiments/mimic_cxr_image/results/ in the SAME JSON/JSONL format as the source, so the cross-dataset comparison table can be filled directly.

Definition of done: the real keyed-run numbers are committed under results/, and the row/cell for MIMIC-CXR image in the cross-dataset table can be read off them. Part of #295.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions