Default Model

Default Model

The single place that states which model DeepDelve treats as its default and as the baseline for evaluations. The project rules (CLAUDE.md, Model Evaluation Standard, point 2) point here instead of naming a model, so a change of model is one edit to this page plus the places listed in section 4. Anything that names a model elsewhere is a copy of this page and may be stale.

Last updated: 2026-10-26. Serving backend for every figure below: Ollama 0.40.9 (native /api/chat).

1. Current state

Role	Value	Status
Shipped default (what a fresh install uses)	deepdelve-gpt-oss (a gpt-oss:20b derived tag)	Superseded by decision, not yet replaced. The user ruled gpt-oss out as baseline on 2026-10-25 after repeated problems (hidden reasoning cost, no true off switch, high run-to-run variance). The shipped files still name it (section 4).
Evaluation baseline for new bake-offs	None designated	A replacement is chosen only after the A/B in section 3 and the user's approval. Do not fall back to gpt-oss.
Leading candidates	ornith-1.5:9b, gemma4:12b	Candidates, not defaults. Evidence in section 2.

Rule in force: no default changes without an A/B and the user's explicit OK.

2. Evidence so far

Comprehension Diagnostic (see Comprehension Diagnostic): 101 reading questions and a writing test on ten papers. Writing, not reading, was the bottleneck for every setup.

Model	Size on disk	Reading (of 101)	Writing defects	Severity 3
ornith-1.5:9b	6.6 GB	95	10.9%	1
MiMo-V2.6-Distill-Qwen-9B	10 GB	94	17.0%	0
gemma4:12b	8.0 GB	91	11.7% (10.3% at 262144 context)	0
Bolt-E4B	6.2 GB	82	11.6%	0
gpt-oss:20b (named level medium)	13 GB	90	20.5%	5

Pipeline path, first live comparison (full write-up and the resulting changes: Pipeline Model Comparison; 2026-10-25, same code and settings, deterministic evidence stage then writer, 20 minute cap, one run per cell, so a signal and not a verdict):

Query	ornith-1.5:9b	gemma4:12b
Congestion pricing	Four near-duplicate facets; writer run stopped by the user, no report to score	Three distinct facets; 393-word report, 4 of 8 sources cited, clean in 3 minutes
Heat pumps vs gas furnaces	1,732 words, 19 of 24 sources cited, no figure absent from sources, several figures cited to the wrong source number, final check unresolved	498 words, 6 of 23 sources cited, hit the cap with the final check unresolved

Free-form agent loop with ornith-1.5:9b failed: the planner's delegation calls were rejected for a missing field and the writer narrated instead of writing, leaving a 77-word recovered draft. Its raw tool-call check passed (5 of 5 valid), so the failure is in the multi-agent role, not the tool format.

3. How a default is chosen

1. Raw API test of the operating mode on the exact served model (thinking control, tool format, usable context).
2. One variable at a time, every other role on the current baseline, no VRAM-forced swaps.
3. Same queries, code and settings for every candidate; at least one fresh non-English query.
4. A discard needs more than one run; a clean pass can stand on one.
5. The user approves before any shipped file in section 4 changes.

4. Where the model name is set

Change all of these together, then update the changelog below.

Location	What it holds
src/tools/config_template.yaml, api.openai_model and the comment above it	The shipped default
README.md, "Default model" and "Model choice" paragraphs, plus the ollama create instructions	User-facing statement of the default
src/pipeline/run.py, the --model default of the evidence stage	Hardcoded copy that does not read the config; should be made to read the configured model
~/.deepdelve/config.yaml	The maintainer's live config (not versioned)
eval/*.yaml candidate configs	Per-candidate settings; these name candidates on purpose

5. Operating facts per model

Thinking control is read from Ollama (/api/show) by src/utils/thinking.py; callers state an intent and the module picks a value valid for the model. The kinds below were raw-tested.

Model	Thinking control	Notes
gpt-oss:20b	Levels (low, medium, high), no off mode	Boolean false sends no reasoning instruction at all; set settings.thinking_effort
ornith-1.5:9b	Boolean (no listed values)	false is off; the string low turns thinking on
gemma4:12b	Boolean (listed false, true; default true)	false is off
granite4.2:8b	Capability only, no listed values	Treated as boolean; not raw-tested for this

settings.enable_thinking stays false. settings.thinking_effort on a boolean model raises an error; a level model without it logs a warning.

6. Changelog

• 2026-10-26: page created. gpt-oss recorded as superseded, no replacement designated; candidates and first pipeline comparison recorded.