Model Bakeoff

Model Bake off

The full local model bake off history and verdicts, moved from MODELS.md. 21 candidates tried against DeepDelve's real multi agent roles, evaluated on two live benchmarks. gpt-oss:20b is still the only one with a full pass on both. Passing an isolated tool call smoke test is never sufficient, every candidate here was run through the actual pipeline. Full evidence trail lives in the Changelog; this page is just the verdict summary. Verdicts follow the project's own Model Evaluation Standard in ROADMAP.md: a discard needs more than one run, and serving layer bugs get named explicitly rather than blamed on the model.

Passed

gpt-oss:20b (~13GB). The default. 7/10 on Colombia B2B, real grounded reports on sales forecasting re runs. High run to run variance but bad runs are honest, not fabricated. One standing gap: on multi facet queries it reliably abandons the harder half rather than fabricating, confirmed even after every pipeline bug was fixed, a genuine model capability limit, not a bug. Its own chain of thought stays cleanly isolated by Ollama, so it doesn't pollute output.

Disqualified

Candidate	Size	Verdict
qwen3.6 (35b a3b)	~23GB	Researches well, reconstructs cited URLs from filenames at synthesis. †
mistral-nemo:12b	~7-8GB	Genuine research happens, but joins the thin_coverage non convergence family on both Ollama and vLLM.
Gemma 4 12B (SetneufPT)	~7.2GB	Reasoning loop near the end, repeated delegation rejections.
Gemma 4 12B (yuxinlu1 fine tune)	~7.4GB	Repeats write_todos with byte identical args even after an explicit stop error, the most severe repetition loop found.
Bonsai 8B (1 bit)	~1.2GB	Research fine, but skips write_workspace_file entirely in writer roles.
qwen3:4b	~2.5GB	Repeats a canned "scope is complete" response instead of acting on nudges. †
qwen3:8b	~8.8GB	Same thin_coverage non convergence, confirmed on both Ollama and vLLM with nothink verified clean.
llama3.2:3b	~2-6GB	Array arguments come back as JSON encoded strings on Ollama, llama.cpp, and vLLM alike, confirmed model side, not serving layer (ollama/ollama#6155 is not the cause).
qwen2.5:3b-instruct	~2-6GB	Writer role never calls the write tool on Ollama; malformed tool JSON on vLLM. Two different failure modes, same verdict.
InternScience/Agents-A1-4B	~5.2GB	Fabricates real sounding, never fetched Wikipedia citations; PeerReviewer approves them anyway. †
qwen3:4b + GRPO fine tune	~4.3GB	The targeted thin_coverage bug is gone, but citation fabrication persists untouched.
qwen3-4b-combined-v2-lora (7 dim GRPO)	~4.3GB	Held out eval improved 0.615→0.781, real generalization, but citation fabrication and writer convergence still broken at 4B even with reasoning cleanly isolated (retested twice, clean the second time).
Ministral-8B-Instruct-2410	~4.9GB	Passes a trivial single arg tool call, fails 3/3 on the real array of tasks schema, narrating JSON as markdown text.
Falcon3-10B-Instruct (community tool tag)	~6.3GB	Real tool calling capability (TII's own training data confirms it), but only ~33% reliable, traced to a known Ollama bug where a tool name mismatch causes a silent empty response instead of an error.
granite3.1-dense:8b, phi4-mini:3.8b	5.0/2.5GB	Both narrate tool calls as literal text despite claimed support.
Phi-4 14B (community tool tag)	~9.1GB	0/9 real tool calls, fully deterministic narration, worse than Falcon3's intermittent failure.
llama3-groq-tool-use:8b	n/a	Rejected at the schema stage.
hermes3:8b	~5GB	Passes the isolated smoke test, but fabricates fictional system error text under DeepDelve's real prompt.
mistral:7b-instruct	~5-14GB	Narrates its delegation call as markdown text instead of a real tool call.
MiniCPM5 1B	n/a	Zero delegation calls across every independent run, both as a paired specialist and full replacement.

Ornith-1.0-9B (deepreinforce ai, MIT license, ~9.5GB) deserves its own note: this closed a long standing inconclusive status, since every prior run had an independent, non model confound (a chat template bug, a native backend corruption bug, two DeepDelve bugs), all now fixed. The clean re test on 2026-08-19 did real research cleanly, 17 sources, 22 findings, no corruption, no loop, but at the delegation quota exhaustion point the Planner narrated its wrap up as chat prose instead of calling the write tool. This is the same "narrate instead of write" failure class seen across most sub 14B candidates tried, and is genuinely the strongest research quality candidate at this size, the ceiling isn't research capability, it's converting research into a written artifact under completion pressure.

Inconclusive, blocked, or not yet viable

Not disqualifications, each hit an infrastructure or fairness blocker before the model's actual capability was cleanly tested.

• qwen2.5-coder:14b-instruct: wrong tool parser on the first attempt (needs <tools> tags, not Hermes), and even with the correct community parser installed, roughly 50% unreliable extraction. Never cleanly resolved either way.
• MiniCPM3-4B: promising on paper (BFCL v2 71.6), but a real infrastructure hang, OpenBMB's own reference tool parser re scans the entire generation on every token, a catastrophic backtracking risk, not a DeepDelve bug. Never reached testing actual research behavior.
• MiniCPM4-MCP: real infrastructure built and kept (finetune/minicpm_tool_proxy.py, a translation proxy for its Python code block tool call format), but the model itself surfaces a different reliability problem each run, not yet a stable specialist candidate.
• Tongyi-DeepResearch-30B-A3B: architecturally a single fine tuned model, not a multi agent system, with no grounding layer comparable to DeepDelve's. Two attempts, both impractical, one killed for being too slow while genuinely making progress, one making zero progress at all despite a faster smoke test.
• ministral-3:8b (~6GB, Mistral's own Dec 2025 core library model, a distinct, newer family from the already disqualified October 2024 Ministral-8B-Instruct-2410): BLOCKED, not disqualified, not passed, by an open, unfixed upstream Ollama bug (ollama/ollama#6351, #12064) that corrupts a tool call's JSON whenever its content contains a raw newline, hit on both the native and OpenAI compat endpoints. Isolated smoke tests were the best of any candidate tried this session (5/5 and 3/3 real structured calls). Two real setup gaps got found and fixed first, a bare Ollama tag silently serving at 4096 tokens instead of this project's 16K floor, and a temperature far above Mistral's own documented ceiling for agentic reliability; fixing both took the score from a crash to 0.750, tying the gpt-oss baseline on one run. But across 5 full live runs, the JSON bug fired in 4 of 5, confirmed independent of context, temperature, and endpoint choice, and a retry-on-failure workaround didn't help either, since the near-deterministic decoding needed for Mistral's own tool calling reliability reproduces the same broken byte sequence on retry. The underlying research quality is genuinely good when a run completes clean. A real path back to vLLM (which has an official Ministral-3 recipe) was researched and ruled out, no pre-quantized checkpoint exists for this model at any size that fits this card, and every fallback path either reopens the ROCm bitsandbytes blocker already closed elsewhere or adds unproven risk. Closed out pending an upstream Ollama fix or a deliberate future AWQ self-quantization effort, not something to retry blindly.

Not attempted

devstral:24b, GLM 4.7 Flash, and Ornith-1.0-35B were all discarded on hardware grounds before any pull, their real footprint (or the smallest available quant) exceeds this card's 17.1GB VRAM budget. A shortlist of narrow function calling finetunes (xLAM-2-8b-fc-r, watt-tool-8B, ToolACE-8B) stays deprioritized, since narrow finetunes overfit to schema correctness at the cost of general instruction following. AgentFloor: How Far Up the Tool Use Ladder Can Small Open Weight Models Go? (arXiv:2605.00334) has since been read in full, see the Literature Review's AgentFloor section for what it found.

Hosted

NVIDIA NIM free tier: best discovery quality of anything tried, but the quota wall kills a run at around 10 minutes.

DeepSeek V4 Flash/Pro: disqualified, both variants, across 4 runs. Two real harness bugs got found and fixed first (a hosted thinking mode control gap, and a guardrail starvation bug in the API server's own budget cutoff), but even with both fixed, DeepSeek re fabricates the same citations on redo, a genuine reliability failure, not an infrastructure artifact. Pro scored worse than Flash despite costing roughly 3x more. Not disqualified for tool calling mechanics or "no write access," both were clean or accurate.

Cross cutting note: the Qwen3 think mode bug

Every row marked with a dagger above was very likely benchmarked with uncontrolled reasoning leaking into its actual output, not the clean output its score implies. Neither the OpenAI compat disable flag nor Ollama's own native think: false field suppresses Qwen3's reasoning, and native think: false is actively worse than doing nothing, dumping raw chain of thought directly into message.content with no separate field, while think: true correctly isolates it. This is confirmed to be an Ollama bug, not a Qwen3 limitation, since the same model on a real vLLM server gives clean, unpolluted output. api.backend: "ollama" (native /api/chat) now gives the same clean behavior without a full backend swap, live verified for gpt-oss and Ornith-1.0-9B. Only qwen3-4b-combined-v2-lora has been re benchmarked through this clean path so far; the rest of the dagger marked rows still stand on their original, likely polluted scores. See the Changelog for the full trace.

The meta result across every run and model: no fabricated report has ever gotten past the grounding gates unlabeled. The defense layer is the validated product; model quality only determines how often it has to fire.