Benchmark - Oncology Decision Boundary Benchmark (ODBB)

Companion repository for "Forty-two percent of oncology decision points defeat all frontier LLMs: mapping a collective capability boundary in clinical reasoning" (Zhang et al., npj Digital Medicine, 2026).

What this contains

• scorer/ - the deterministic scoring engine with 14 typed failure labels (no LLM-as-judge, bit-for-bit reproducible)
• pipeline/ - the benchmark generation pipeline (NCCN guideline workspaces and colorectal cancer case reports to structured decision items), including a packaged build-time skill (pipeline/skill/) and workflow documentation (pipeline/workflows/)
• docs/ - method documentation: failure taxonomy, question-type taxonomy, scorer design, reproducibility, clinician adjudication protocol
• data/ - released upon publication: the raw outputs of all nine evaluated models, the clinician-adjudication and gold-audit records, the per-item CRC defect flags, and the scorer-calibration pilot (available to referees during review). The 1,586 NCCN-derived benchmark items are not redistributed as open data (derived from NCCN copyrighted guidelines; academic/research use only, redistribution subject to NCCN's terms) and are regenerable via the released pipeline/
• analysis/ - paper figure/table reproduction scripts, including revision_2026/ (contamination/recency, scorer threshold-sweep, gold-answer audit, and CRC-cleaning reproducibility scripts from the revision round)
• examples/ - end-to-end usage examples

At a glance

• 2,005 scorable clinical decision points (1,586 NCCN + 419 CRC)
• 14 typed failure labels, fully deterministic scorer (zero LLM-as-judge)
• 9 frontier LLMs evaluated
• 2 waves of independent clinician validation (225-item + 80-item)

Quickstart

pip install -r requirements.txt
pytest scorer/tests/

For end-to-end scoring against the benchmark items, see examples/.

Citation

If you use this benchmark or the scorer, please cite:

Zhang S., Li J., Chen W., Bao Z., Wang Y. "Forty-two percent of oncology decision points defeat all frontier LLMs: mapping a collective capability boundary in clinical reasoning." npj Digital Medicine (2026).

See CITATION.cff for machine-readable citation metadata.

License

• Code (scorer/, pipeline/, analysis/, examples/): Apache-2.0 (LICENSE-CODE)
• Data (data/): CC-BY-NC-4.0 (LICENSE-DATA)

Contact

Corresponding authors: zwbao1996@zju.edu.cn, yosean4desci@gmail.com