Writer Redesign Plan

Writer Redesign Plan

The working plan for the writer redesign (branch writer-redesign), written 2026-10-26 so that no proposal is lost between sessions. It follows from the Comprehension Diagnostic (writing is the bottleneck), the literature review and the Pipeline Model Comparison. Test design and decision rules are in the Writer Role A/B Protocol; the model question is on the Default Model page. Status is updated in place.

Phase 0: settle the design

Item	Status
A/B protocol: arms A0 to A4, ornith-1.5:9b and gemma4:12b, three seeds, defects counted over model-written units only, fixed pass rule	Done, awaiting review
Command-line entry points read the configured model instead of a hardcoded one	Done
Runners for arm A1 (prompt variant) and A2 (schema-restricted citation ids)	Open

Phase 1: build behind a flag

Item	Status
Compiled writer (src/writer/compiled.py): code renders quotes, figures, citation numbers, gaps and sources; the model writes short verified sentences; best of N, one repair, verbatim fallback	Done, offline tests pass
writer_mode setting on the CLI, TUI and API through the one shared evidence loader	Done
Thinking control derived from the model	Done, merged to main
Quote check made link-aware, matching the pipeline's own verification	Done, regression test
Safeguards: quotes fenced as untrusted data, prompt size cap, bot-wall titles replaced	Done, tests
Live smoke run through the real completion checks	Running; first result: one cross-source contradiction check handed the report to an agent rewrite
Citation-number mapping of the figure-context advisory on compiled reports	Open (probable false alarms)

Phase 1b: make the compiled path deterministic

Completion checks fall into four classes, and each class gets one policy.

Class	Checks	Policy
Satisfied by construction	Missing artifact or findings, non-URL citation, no citations, uncited claims, paraphrased quote, unsupported figure or regulation identifier, stub source, duplicate sections	Assert as invariants; a firing is a compiler bug and fails loudly
Code can render the fix	Cross-source contradiction, undisclosed evidence gap, report underuses findings	Reuse the check's detector and render the block in code
Statistical	Claim unsupported, NLI, topical mismatch, editorializing, meta-language	Measure false positives on pure quotes
Needs research or a better plan	Missing query facet, missing specific item per facet, thin coverage	Render a visible "not covered" caveat

Result of the first step (2026-10-26): the skeleton replay compiled a worst-case report (every unit a verbatim quote) from each of 25 saved evidence directories and ran the model-free checks. It started at 14 of 25 clean and ended at 25 of 25 after fixing the causes, which were all inconsistencies between the checks and the pipeline's own quote verification, not model behaviour: link syntax and footnote markers handled differently, magnitude suffixes in figures, a page's own reference numbers read as citations to our sources, gap statements treated as claims, and a quote already proven verbatim being out-voted by a term-overlap heuristic. One real class remained, sources that disagree on a figure, now rendered in code as a "Where sources differ" block. The checks are shared with the agent path, so those fixes also reduce false alarms there. Still open: the replay covers only the model-free checks, and no real-model replay has been run yet.

Evidence that it works, none of it needing a GPU: a skeleton replay (a stub model that always falls back to verbatim quotes, run through the model-free checks on all saved evidence directories, producing a firing matrix), the same for real-model reports on a few directories, and a termination rule (one deterministic handler pass per verdict, never an agent dispatch; unresolved verdicts become a visible unverified banner).

Phase 1c: context and poisoning

Seen in real runs: a warm-climate cost quote kept for a cold-climate question, captcha pages saved as sources, a bot-wall title on a good page. Planned: an injection red-team test (plant instructions in saved sources and measure survival through extraction, judge, writer and verifier), a scope-aware relevance judge, corroboration for figures in the overview, source tiers shown by code, a context card per quote, and a decision on whether the Evidence list shows all quotes or only cited ones.

Phases 2 to 4

Phase	What	GPU
2	Stage 1 A/B on the ten keyed sets. Done 2026-10-26, see results: compiled writer passes for ornith, no clear gain for gemma, prompt-only arms fail coverage	Yes, about an hour per model
3	Winning arm in the real loop on three to four full queries; finish the remaining comparison queries (semaglutide, Iceland 2008, Spanish sugar taxes) with at least two runs per cell	Yes
4	Default-model decision, update of the shipped defaults and README, merge with the user's approval, parity and verdict-matrix checks	No

In parallel

Planner duplicates (test gemma4:12b and MiMo as planners on the 21 saved queries, or offer plan review; no dedupe code is kept after four failed attempts), ornith's failure in the free-form agent loop, and the thinking resolver's blind spot for a specialist model that differs from the main model.