Pipeline Model Comparison

Pipeline Model Comparison (2026-10-25)

First live comparison of two replacement candidates for the shipped default (see Default Model), run on the deterministic evidence path introduced by the stage-graph pipeline. It follows the Comprehension Diagnostic, which found that writing, not reading, is the bottleneck, and it is the bridge from that study to the changes listed in section 5.

1. Setup

• Models: ornith-1.5:9b and gemma4:12b, one at a time on a single 16 GB GPU, Ollama 0.40.9, native endpoint, thinking off (resolved per model by src/utils/thinking.py).
• Path: the evidence stage (python -m pipeline.run: one planning call, code search and fetch, BM25 and cross-encoder extraction, one judge call per facet) followed by the writer roles through --pipeline-evidence. The same model served both stages.
• Held constant: code (main at the time), settings, 64K context for the writer, hard cap of 20 minutes per writer run.
• Excluded by decision: gpt-oss:20b, no control arm.
• Queries run: two English queries. Three further fresh queries (one of them Spanish) were planned and not run; the session ended at the first query for time.

2. Evidence stage

Query	Model	Facets	Facets answered	Quotes kept	Sources fetched	Time
Heat pumps vs gas furnaces	gemma4:12b	4	4	54	77	218 s
Heat pumps vs gas furnaces	ornith-1.5:9b	4	3	65	80	204 s
Congestion pricing, traffic and air quality	gemma4:12b	3	3	19	30	about 100 s
Congestion pricing, traffic and air quality	ornith-1.5:9b	4	3	52	69	180 s

Both models gathered usable evidence in a few minutes. The difference is in planning: on the congestion query ornith produced two pairs of near-duplicate facets (traffic volume and traffic congestion, air pollution and emissions) and nothing on cost, while gemma produced three distinct facets including economic impact. On the heat-pump query both models planned the same fair four-way split.

3. Writer stage

Query	Model	Words	Sources cited / fetched	Attempts	Wall time	Final state
Heat pumps	ornith-1.5:9b	1,732	19 / 24	3	15 min	Unresolved check (untracked delegation)
Heat pumps	gemma4:12b	498	6 / 23	6	20 min (cap)	Unresolved check (report underuses findings)
Congestion	gemma4:12b	393	4 / 8	3	3 min	Clean
Congestion	ornith-1.5:9b	not run to completion				Run stopped by the user

No figure in any finished report was absent from every source. Ornith's heat-pump report answered both halves of the question, had analysis and evidence-gap sections, and disclosed its one gap. Its defect was citation numbering: several figures were attached to a source number other than the source that contains them, and five listed sources were never cited. Gemma's heat-pump report was short and left four listed sources uncited, and the completion loop (six repair attempts) never converged before the cap. Four of the sources ornith cited were company blogs.

A separate trial of the free-form agent loop (the Planner delegating to sub-agents) with ornith failed: delegation calls were rejected for a missing field, the writer narrated instead of calling the write tool, and the result was a 77-word recovered draft citing none of 18 fetched sources. Its isolated tool-call check had passed (5 of 5 valid), so the failure is in the multi-agent role.

4. Planner duplicates: what was tried

Four ways of removing duplicate facets were tested on the 21 saved plans and none was kept:

Approach	Result
Embedding similarity threshold	Cannot separate the duplicate (0.92) from legitimate neighbours (Finland vs Estonia 0.86, Phase 2 vs Phase 3 trial 0.93)
One call that groups duplicate facets, then merge	Catches the congestion duplicates but merges facets that measure different things (leave days with pay calculation, LSM trees with B-trees)
Pairwise judgement that must state the difference first	Never merged wrongly, but also missed the real duplicates
Prompt rule, one facet per distinct thing asked	Cut facets from 77 to 66 across the queries without discrimination, and the target pair survived

The deciding information is semantic (volume overlaps congestion; days do not overlap pay), and a 9B model gets it wrong in both directions. Gemma did not produce the duplicates on the one query where ornith did.

5. Conclusions and resulting changes

Conclusion	Change	Status
The pipeline hard-coded a gpt-oss thinking level; on ornith it silently turned thinking on	Thinking control derived from the model (src/utils/thinking.py), pipeline asks for "minimal"	Done, 891bdaa
Ornith writes fuller, better-covered reports but attaches figures to the wrong source number; gemma cites few sources and its repair loop does not converge	Writer redesign: code renders figures, quotes and citation numbers from the evidence table; the model writes only connectors and the evidence paragraph (hypotheses H1 to H4 in The Writer Role)	Next build, behind a flag; tested first on the ten keyed findings sets
No reliable automatic dedupe of facets at 9B	No dedupe code. Test gemma4:12b and MiMo as planners on the 21 saved queries; offer plan review (human_in_the_loop)	Open
Ornith fails in the free-form multi-agent loop	Do not use ornith for the free-form path until the delegation-field rejection is understood	Open
Pipeline --model default still names gpt-oss and ignores the config	Make it read the configured model	Open
One query per model is not a verdict	Run the three remaining queries (one Spanish) with the same driver, at least two runs per cell before any discard	Open
gpt-oss is out as baseline; no replacement is proven	No change to shipped defaults until the writer A/B passes and the user approves	Rule in force

6. Threats to validity

One run per cell on two queries; the writer's repair loop is nondeterministic in length, so the 20-minute cap truncated one run; the scorecard is automatic and its citation checks are advisory; the same model served both stages, so a planner effect and a writer effect are not separated; ornith's congestion writer run was stopped, so that cell is empty; no gpt-oss control by decision.