Writer Role AB Protocol

Writer Role A/B Protocol (stage 1)

Pre-registration for the first test of the writer redesign described in The Writer Role. Written 2026-10-26, before any output of the new arms exists. Stage 1 isolates the writer by giving every arm the same true-by-construction findings; stage 2 (section 6) tests the winning arm inside the real loop. Decision rules below are fixed now and are not changed after seeing results.

1. Question

Does a writer that takes figures, quotes, citation numbers and gaps from code, and lets the model write only short connecting sentences, produce fewer defective claims than the writer as shipped, without lowering coverage?

2. Inputs, models, seeds

• Inputs: the ten findings sets of the Comprehension Diagnostic (hand-keyed answers, true by construction), so every defect belongs to the writer.
• Models: ornith-1.5:9b and gemma4:12b, one at a time, thinking resolved per model (src/utils/thinking.py), 64K context. gpt-oss is excluded by decision.
• Seeds: three per cell (0, 1, 2), the baseline arm included (fresh runs, so every cell shares the same code, settings and serving).
• Raw check first: thinking control and a schema-constrained call are raw-tested on each served model before its cells run (Model Evaluation Standard, point 1).

3. Arms

Arm	What changes	Implementation
A0	Baseline: the Builder instructions as shipped, free text, seeds 0 to 2	eval/comprehension/run_writing.py with DD_SEED
A1	H3: task prompt asks for facts, caveats and gaps only, forbids new figures, names and abbreviation expansions, fixes a length budget	Prompt variant of A0
A2	H2: A1 plus citation ids restricted to the finding numbers by a JSON schema, evidence field before text	eval/writer_spike pattern, per finding numbering
A3	H1: code renders headings, quotes, citation numbers, gaps and sources; the model writes only the overview and one short paragraph per topic; each sentence is verified	src/writer/compiled.py, one best-of-1 draft, one repair, verbatim fallback
A4	H4 on A3: best of three drafts, first clean draft wins	src/writer/compiled.py with n_drafts=3

H5 (separate reader and writer) is deferred to stage 2. In the findings sets each finding becomes one numbered quote, so citation validity can be measured for A1 to A4; A0 cites the single paper and cannot be compared on that measure.

4. Measures

Primary. Defects per report and severity 3 defects per report, using the diagnostic's labelling rules (eval/comprehension/units.py for units, same defect types and severities). Denominator rule (set in advance because it matters): a code-rendered quote bullet is literal by construction and is verified by substring match, not labelled. Rates are computed over model-written units only. Without this rule, pasting quotes would lower the defect rate by dilution alone.

Deterministic, per report. Invalid citation ids, figures (two or more digits) absent from the findings, abbreviations absent from the findings, uncited sentences, words, seconds, and the share of units that fell back to a verbatim quote.

Also. Coverage of the query's parts (each keyed question answered or its gap stated, scored as in the reading test), and a readability rating by one rater, recorded separately and not part of the pass rule.

Labelling plan (added 2026-10-26, before any stage 1 output beyond a one-paper dry run). All three seeds are generated and every deterministic measure is computed on all of them. Defect labelling is done on seed 0 first: 5 arms by 10 papers by 2 models, 100 reports, with arm and model hidden. Seeds 1 and 2 are labelled only if the decision rules of section 5 cannot be applied to seed 0 alone, that is when an arm's Wilson interval overlaps the pass threshold or the baseline cell. This keeps the labelling effort proportional to what the decision needs and fixes the rule before results exist.

Labeller. One labeller, as in the diagnostic. The labelling key is written before the new arms run, and labelling is done blind to the arm.

5. Decision rules

An arm passes stage 1 for a model when all of the following hold on the pooled three seeds:

1. Defect rate over model-written units is below the lowest rate the shipped writer reached in the diagnostic (15.4 percent), and below the arm's own A0 cell for the same model.
2. Zero fabricated citations: no invalid ids and no sentence whose cited quotes do not contain its figures.
3. No coverage drop against A0 for the same model.
4. Severity 3 defects per report not above A0's.

Intervals are Wilson 95 percent over units; with ten papers they are wide, so a clean pass can stand on one pooled result, but a discard needs a second run (Model Evaluation Standard, point 4). A tie on rule 1 is broken by the lower fallback share, then by seconds.

6. Stage 2 and what follows

If an arm passes, it runs inside the real write, review, fix loop on three to four full queries through the deterministic pipeline path, scored with scripts/evaluate_run.py and compared against the shipped writer on the same evidence. The remaining comparison queries of Pipeline Model Comparison serve as that set. Merging and any change to the Default Model need the user's approval.

7. Blast radius to check before an arm touches the engine

The completion-check verdict matrix (tests/test_structural_checks.py), the FindingsWriter and compaction interplay, citation detection in utils/grounding.py, the new setting in run_cli, the TUI and api.py and in the save_config allowlist, and the research-remedy checks that evidence-only runs skip.

8. Changelog

• 2026-10-26 (later): stage 1 run; results and the post hoc arms A3b and A4b (overview prompt fixed after seeing seed 0) are on Writer Role A/B Results. Labelling deviations: units counted with a corrected sentence rule, seed 1 labelled only for cells that overlapped a decision line.
• 2026-10-26: protocol written; arms A3 and A4 implemented and unit-tested offline; one live smoke run of the compiled writer on a saved evidence directory (36 s, 5 units, 1 repair, no fallback) is the only model output seen so far.