Repository navigation
Data and analysis code accompanying:
Auditing the Human–LLM Autonomy Gap in Clinical Ethics: Development and
Application of the Autonomy Index Across 50 Clinical Ethics Vignettes
Taposh Dutta Roy, Rebecca Weintraub Brendel
Center for Bioethics, Harvard Medical School
medRxiv preprint (2026). DOI: [add once posted]
Contents
- Instrument — the 13-item Autonomy Index as presented to human reviewers
and to models, covering Values Awareness, Factual Understanding, Rational
Deliberation, and Intentional Action on a common 0–4 ordinal scale, plus the
two External Constraint and Support Presence items. - Human ratings — de-identified item-level and vignette-level ratings.
173 ratings across 50 vignettes from 30 reviewers, collected February to
June 2026 under a rolling recruitment protocol. - Model ratings — item-level responses from eight frontier models, ten
independent runs per model per vignette (4,000 attempted, 3,431 valid
parsed responses) collected in July 2026. - Vignette corpus — the 50 comparison vignettes with their derivation from
the 89-vignette March pool, including the score-tier and disagreement
criteria used for selection. - Analysis code — scripts reproducing the estimates, tables, and figures
in the paper, including the prorated item-level pooling estimator, the
variance decomposition, and the Overton coverage computation. - Prompt and model configuration — the full prompt as issued, with model
version identifiers, temperature, and provider routing settings.
Data were frozen for analysis on 26 June 2026. Submissions received after that
date are not included in this release.
Not included
Individual reviewer identifiers and registration data are not shared.
Participants consented to aggregate research use; they did not consent to the
release of individual-level records. Reviewer role, experience band, and
specialty are reported only in aggregate, as in Table 4 of the paper.
Ethical approval
Collected under Harvard Medical School IRB protocol IRB26-0146 with documented
electronic consent.
Known limitations
The human reference standard is based on a convenience sample with an
opportunistic rating allocation, which prevented a stable reliability
estimate. The 50 comparison vignettes were selected on March model scores and
inter-model disagreement, so all estimates are conditional on that selection.
See Sections 9.3 and 10 of the paper.
Citation
If you use these materials, please cite the preprint. A CITATION.cff file is
included in the repository root.
License
Code: MIT. Data and instrument: CC BY 4.0.