Writer Role AB Results

Writer Role A/B Results (stage 1)

Results of the test pre-registered in the Writer Role A/B Protocol, run 2026-10-26 on the ten keyed findings sets of the Comprehension Diagnostic. Every defect belongs to the writer because the input is true by construction. Serving: Ollama 0.40.9, 64K context, thinking resolved per model, three seeds generated per cell. Status of the redesign is on the Writer Redesign Plan.

1. What was labelled

Seed 0 of every arm (100 reports) was labelled blind to model and seed, by one labeller, with the diagnostic's taxonomy and the denominator rule (code-rendered quote bullets and the source list are not labelled). Arm was hidden but a compiled report's headings reveal its format. After the first result a prompt bug in the compiled writer was fixed and two post-hoc arms were added (A3b, A4b); their seed 0 and a seed 1 for the cells that overlapped a decision line were labelled in separate packet sets. Seed 1 reports identical to their seed 0 twin (compiled arms at temperature 0) inherited that label, the rest were labelled by content. Unit counting uses a corrected sentence rule (a full stop followed by citation markers ends a sentence), applied to every arm.

2. Defect rate over model-written units (Wilson 95%)

Model	A0 shipped	A1 prompt	A2 prompt + schema ids	A3 compiled	A4 compiled x3	A3b fixed	A4b fixed x3
gemma4:12b	14.6% (11.6 to 18.2)	2.1%	2.5%	30.9%	31.1%	13.5% (8.4 to 21.1)	13.5% (8.4 to 21.1)
ornith-1.5:9b	24.9% (20.2 to 30.3)	6.5%	1.0%	12.5%	9.3%	5.0% (2.0 to 12.2)	4.4% (1.7 to 10.8)

A0 for gemma and A3b and A4b for both models pool seeds 0 and 1; the other cells are seed 0. The rule-1 threshold is 15.4% (the lowest rate the shipped writer reached in the diagnostic).

3. Findings

• Compiled writer, ornith: passes. A3b and A4b are clearly below both the 15.4% threshold and ornith's own baseline (24.9%), with no invalid citation ids, no figure absent from the findings and no severity 3 defect. Ornith's baseline is the worst cell; the redesign removes most of its defects. By the protocol's tie-break (lower fallback share, then seconds) A4b edges A3b (23% against 33% of units falling back to a verbatim quote) at about 1.5 times the time.
• Compiled writer, gemma: no clear gain. After the fix gemma A3b and A4b are 13.5% against a baseline of 14.6%; the intervals overlap each other and the threshold. Gemma's baseline is already near the threshold, so there is little for code to remove, and best of three adds nothing here (0 fallbacks in either).
• The first compiled result was a prompt bug, not the approach. 17 of gemma A3 and A4's 21 defects were one sentence in the overview unit ("the provided quotes do not state what the study leaves unanswered", or a stated finding presented as unanswered). The fix asks the overview only for findings, forbids describing the quotes and adds a verifier rule; it is post hoc, so the registered A3 and A4 verdicts stand as measured.
• Prompt-only arms have the lowest defect rates and fail coverage. A1 and A2 write about half the words of the baseline. By a crude proxy (keyed numbers present in the report) recall falls from 78% to 71% and 60% for gemma and from 100% to 87% and 92% for ornith. They fail rule 3. A defect rate this low also reflects reports that say little; the proxy is not a labelled coverage score.
• Citation validity is not the issue in this setting. Invalid ids are zero in every arm and absent figures occur only in ornith's baseline (5 reports) and A1 (1). The single severity 3 defect is a baseline ornith report comparing a cross-sectional odds ratio with a hazard ratio, the trap the findings warn about.
• What defects remain in compiled reports: a hedge dropped when a finding says the authors infer a cause; a result reported without the note that no significance test exists; a quoted result presented as the study's own when it comes from cited work; a stated limitation presented as an unanswered question; text and table figures merged ("or") where the findings say they differ. All are in the model-written overview and findings sentences.
• Fallback share is high for ornith (23 to 35%): a quarter to a third of its units end as a verbatim quote because the verifier rejects the draft. Those units are true by construction but read as quotes.

4. Decision under the registered rules

Arm	Model	Verdict
A3, A4 (as registered)	gemma	Fail rule 1
A3, A4 (as registered)	ornith	Rule 1 met, interval overlaps the threshold; superseded by A3b and A4b
A3b, A4b (post hoc)	ornith	Pass stage 1 on defect rate, citations, figures, severity 3, coverage (quote bullets keep every finding)
A3b, A4b (post hoc)	gemma	Not shown better than the baseline
A1, A2	both	Defect rate met, coverage dropped, fail rule 3

No default changes. Stage 2 (the winning arm inside the real loop on three to four full queries, scored with scripts/evaluate_run.py against the shipped writer on the same evidence) is the next gate, and the choice of default model still needs the A/B and the user's approval (Default Model).

5. Threats to validity

One labeller, not blind to arm in effect (format visible), and A3b and A4b were designed after seeing seed 0 of A3 and A4; their seed 1 labels are the only fresh data. Compiled arms are near-deterministic at temperature 0 across seeds, so extra seeds add little for them and mostly test the baseline. The coverage proxy counts keyed numbers only. The ten findings are one per answer from single papers, easier than the evidence the real pipeline produces (many sources, duplicates, poisoned quotes), so a pass here says the compiled writer does not add defects on clean input, not that it holds on real evidence. Defect labels are in the local eval/writer_ab/out/ directory (not committed; reports quote paper text).

6. Changelog

• 2026-10-26: stage 1 run, seed 0 labelled for 100 reports; overview fix; A3b and A4b labelled for seeds 0 and 1; gemma baseline pooled over seeds 0 and 1.