MedIWF v1.0.4

Version 1.0.4 (22 October 2026)

• Independent reader check of the 3 most frequently flagged rules (protocol amendment 7, 21 October 2026, with its addendum of 22 October): 2 readers not involved in the study rated a flag-enriched sample of 120 corpus items. Files: docs/protocol_amendment_7.md, scripts/build_independent_check.py, scripts/analyze_independent_check.py, scripts/independent_check_weighted.py, outputs/independent_check/ (sample, instruction sheet, ratings with the detector's flags, results and 2 sensitivity analyses), outputs/independent_check_weighted_v1.txt. The readers' returned spreadsheets are not released.
• scripts/kappa_reanalysis.py and outputs/kappa_reanalysis_v1.txt: Cohen's kappa with the repository's method for the physicians' acceptability dichotomies, the LLM judges against the physicians on the published composites, on shared criteria over the same items, and with each generator judge's own items excluded, and the reader check.
• scripts/arms_alphabetization_summary.py and outputs/arms_alphabetization_summary_v1.txt: answer-key position by alphabetization under each prompt, the LLM-NBME gap with and without the longest-option rule, the reasoning and guided-neutral arms against their controls.
• outputs/analysis_verification_prerevision_v1.txt and outputs/verification_sample_pass1_firstlabels.xlsx: the hand-check estimates with the 3 word-repetition labels of rows 16, 143 and 215 at their first value (docs/hand_check_pass1_log.md).
• CITATION.cff: version 1.0.4.

Full list in CHANGELOG.md.

MedIWF v1.0.3

Documentation only. Changes since 1.0.2 (details in CHANGELOG.md):

• docs/errata_2026-09-24.md: note 4 added, recording where the generative-AI disclosure planned in Amendment 6 is made.
• README.md: the Withheld section names the internal review reports, manuscript drafts, screening-test items and scratch backups that the dated records in docs/ cite by path.
• CITATION.cff: version 1.0.3.

No code, data or output file changed.

MedIWF v1.0.2

Corrections to the documentation, the data-package builder and the detector's flag count. Changes since 1.0.1 (details in CHANGELOG.md):

• CITATION.cff: license is a single string again (MIT; the data license is described in LICENSE-DATA.md), so that the citation metadata can be read by archives; version 1.0.2.
• scripts/build_paper_data.py: for an analysed attempt the item text is taken from the re-parse the detector used; in the paper's Dataset 1, exactly 18 rows marked analysed = True that had empty item text gain their text, every other cell is unchanged.
• src/mediwf/detector.py: the n_flaws column of outputs/flags_*.csv counted the 14 rules plus the 2 feature checks; it now counts the 14 rules. The 6 flag files of the analysed runs were regenerated and only that column changed; no analysis script reads the column.
• scripts/independent_nbme_check.py and its output: 181 checks (162 in 1.0.1); 179 match, 2 differences explained in the output.
• docs/errata_2026-09-24.md: note 1 rewritten, note 2 names both protocol passages, note 3 added (the combined-prompt judge run of 14 September).
• README.md: the reproduction paragraph lists every non-identical file; the record fields of data/generated/ described exactly. DATASET_CARD.md names the judge_* files; data/external/README.md gives the exact path of the BenchMarker label file.

MedIWF v1.0.1

See CHANGELOG.md. README and reproduction commands re-tested in a fresh environment; license notices for the BenchMarker prompts and the NBME-derived aggregates; errata to the dated documents; independent NBME check script and output; judge validation v2; per-item rating times removed from the public rating files.

MedIWF v1.0

First public release of MedIWF, the corpus and code behind "Item-writing flaws and answer-position bias in large language model-generated medical multiple-choice questions" (Gundam R., 2026).

Contents: 28,280 generated single-best-answer items from 15 large language models across 8 specialties and 200 topics, with prompts and generation metadata (data/generated/, gzip-compressed); the rule-based detector for 14 structural item-writing flaws (version 1.2) with its 65 unit tests; the generation, detection, validation and analysis scripts with their outputs; the physician content ratings of a 100-item sample; the hand-check labels and procedure log; the study protocol with its six amendments; and the supplementary tables.

Licenses: code MIT; data and documents CC BY 4.0. The NBME items used for the psychometric comparison are covered by a data-use agreement and are not included; the aggregate statistics needed to reproduce the figures are.

See README.md for how to reproduce the analyses without network access, and DATASET_CARD.md for the intended use and limitations of the corpus.