Model Bakeoff

Model Bake off

This is the full local model bake off history and verdicts, moved from MODELS.md.

It's a summary of every local candidate tried against DeepDelve's real multi agent roles (Planner, Searcher, Analyzer, Builder, FindingsWriter), evaluated against two live benchmarks, a 13 run Colombia B2B rubric (eval/colombia_b2b_benchmark.md) and a sales forecasting and heuristic algorithms rubric (eval/sales_forecasting_benchmark.md). The bar is the same throughout: passing an isolated tool call schema test isn't sufficient evidence a model behaves reliably in the real role, so every candidate below was run through the actual pipeline, not just a smoke test. The full evidence trail, live run detail, and ongoing trials live in the Changelog; this page is just the current state summary. Verdicts follow the project's own Model Evaluation Standard (see ROADMAP.md's top level section): a discard needs more than one run, and backend or serving layer bugs are named explicitly rather than blamed on the model.

This got moved out of README.md into its own file on 2026-07-29, because the per candidate detail had outgrown what a plain Markdown table can render legibly, each row's notes were paragraph length and wrapped badly in table cells. The format here is one section per candidate, grouped by verdict.

As of this writing, 21 candidates have been tried. gpt-oss:20b is still the only one with a full benchmark pass on both standing benchmarks.

Passed, the default

gpt-oss:20b. Around 13GB of VRAM. Best result: 7/10 on Colombia B2B, and a real grounded report on every sales forecasting re run. This is the default, and the only candidate with a full benchmark pass on both standing benchmarks. Run to run variance is high, but bad runs are honestly empty rather than fabricated, and a run takes about 15 to 20 minutes. Its own chain of thought can't be fully disabled either (see the Qwen3 family think mode note further down for the general issue), but Ollama keeps it in a separate reasoning field that DeepDelve's client never reads as the model's actual output, so it's harmless here.

There's a standing, still open content quality gap, reconfirmed live on 2026-07-31 after that session's completion pipeline starvation fixes shipped. On the multi facet sales forecasting benchmark, the model reliably abandons the harder half of a two facet query rather than fabricating or refusing, the same pattern first logged on 2026-07-14/18 (it dropped the Colombia cultural context half that session), and it was still happening after every structural pipeline bug found that night got fixed. The 2026-07-31 run's final_report.md was real, honestly caveated, and correctly grounded, no fabrication, but it delivered neither the query's own "top 5 heuristic algorithms" ask nor any Colombia cultural pattern integration. The Colombia and heuristic algorithm sources it had fetched just sat unused in the References list. check_report_underuses_findings/_evidence correctly flagged this on every attempt, and the model still hadn't fixed it by the time the wall clock ran out. This is a genuine model capability gap, not a pipeline bug: the completion check machinery is working exactly as designed (real convergence, no starvation, honest output), and the gap it keeps correctly flagging is the model's own unwillingness or inability to actually act on "cover every facet" feedback within a bounded retry budget. See the Pending section of ROADMAP.md for the literature research angle on multi facet task abandonment under iterative self correction.

A follow up live test the same night tried an explicit edit_workspace_file routing directive fix (commit 67e4b00, the smallest literature grounded attempt available). It changed Builder's tool choice correctly, but didn't fix the underlying gap: a subsequent re test's report went from answering roughly a third of the query to answering zero of the ML/heuristics half (100 percent Colombian festivals content, the deep learning facet dropped entirely), despite the directive explicitly saying not to touch any other part of the report. That confirms this is genuinely the self correction blind spot, or aggregator noise, that the literature describes, not a tool choice gap. Prompt level fixes have hit their ceiling here; a per facet Builder dispatch is the justified next attempt (see the Pending section of ROADMAP.md).

Disqualified

qwen3.6 (35b a3b). Around 23GB of VRAM. Best result: 1/10. It researches well but synthesizes disastrously at scale, reconstructing all 22 of its cited URLs from filenames rather than the real ones. (Marked with a dagger below, see the Qwen3 think mode note in the cross cutting section at the end.)

mistral-nemo:12b. About 7.1GB on Ollama, or roughly 8.3GiB on vLLM with bitsandbytes 4 bit. Best result: 2/10 on Ollama, and Report: NOT WRITTEN on vLLM (thin_coverage). Disqualified on both backends. The original Ollama score stood unconfirmed for a while after a real vLLM infrastructure block (chat_template_kwargs rejected by Mistral's native tokenizer, since fixed via settings.skip_chat_template_kwargs). Retested with that fix, the model showed genuine engagement this time (13 sources, 0/7 search failures, 8 findings), but it joins the thin_coverage non convergence family already seen in qwen3:4b/qwen3:8b: all 4 completion check attempts hit the identical problem, the retry budget ran out, and no report ever got written.

Gemma 4 12B, SetneufPT/Gemma4-12B-IT-QAT_Q4_64K_16GB-GPU. About 7.2GB. Best result: Report: NOT WRITTEN. Disqualified for a reasoning loop near the end and repeated delegate_tasks rejections.

Gemma 4 12B, yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF. About 7.4GB (Q4_K_M). Best result: Report: NOT WRITTEN, run twice, identical failure both times. Disqualified for a different and more severe failure than the other Gemma 4 12B candidate above: it repeatedly calls write_todos with byte for byte identical arguments, a literal repetition loop rather than incremental revision, and it kept going even after the tool explicitly errored with "you MUST summarize what you've done and stop." It was DeepDelve's own anti loop quota that ended the run, not the model recognizing completion. This was confirmed over two runs on different queries, one deliberately simple. It isn't a serving layer bug: nothink mode (which needs reasoning_effort:"none", not chat_template_kwargs.enable_thinking) and tool calling were both confirmed clean via a direct API test beforehand.

Bonsai 8B (PrismML, 1 bit). About 1.2GB. Best result: Report: NOT WRITTEN. Disqualified for a worse reason than the Gemma 4 candidates: it skipped write_workspace_file entirely in writer roles, even though research worked fine.

qwen3:4b. About 2.5GB. Best result: Report: NOT WRITTEN, all 8 retries exhausted on thin_coverage. Disqualified. Real research happens, but the model repeats a canned "research scope is complete" non response instead of acting on the completion check's corrective nudge, a non convergence pattern also seen elsewhere as 10 redundant identical write_workspace_file calls on a trivial query. (Dagger, see the Qwen3 think mode note below.)

qwen3:8b. About 8.8GB (fp8, vLLM). Best result: Report: NOT WRITTEN twice in a row, an identical thin_coverage attempt sequence both times. Disqualified, and double confirmed clean of every prior excuse: retested on vLLM instead of Ollama, with nothink mode and tool calling both verified clean via direct API tests beforehand. It's the same failure class as qwen3:4b: once thin_coverage fires, it repeats the same narrated summary verbatim across retries instead of acting on the corrective nudge. That's four total occurrences of this exact non convergence pattern now recorded across different backends and models.

llama3.2:3b. About 2.0GB on Ollama, or 6.0GB bf16 on vLLM. It fails at the schema stage on both backends. Disqualified, and double confirmed clean of the "maybe it's Ollama's fault" excuse: retested on vLLM with a real gated HF checkpoint, not the Ollama GGUF, using the llama3_json tool parser, and the exact same #6155 stringified array shape reproduced 4 out of 4 times on a completely different serving stack. This is confirmed directly in vLLM's own official docs, under "Known issues" for Llama Models: "The model can generate parameters in an incorrect format, such as generating an array serialized as string instead of an array." That's Llama 3.2's own limitation, not an Ollama specific bug. The original #6155 framing is disproven for this candidate specifically.

qwen2.5:3b-instruct. About 1.9GB on Ollama, or 6GB bf16 on vLLM. Best result: Report: NOT WRITTEN on Ollama (missing_findings), and malformed tool call JSON on vLLM at the schema stage. Disqualified on both backends, for different failure modes each time, not a single cause bug. On Ollama, it passed the schema test cleanly and researched fine, but FindingsWriter never successfully called write_workspace_file across 8 attempts, the same root cause as Bonsai 8B. On the vLLM retest, a nested array delegate_tasks shaped tool call consistently produced genuinely malformed JSON: it closes the arguments object but omits the final closing brace for the outer wrapper. That was confirmed via 5 or more reproductions and direct inspection of the parser's own extraction, the parser is correct, and the model's own JSON is broken at the source, not a token budget cutoff.

InternScience/Agents-A1-4B. About 5.2GB (Q8_0 GGUF, Ollama). Best result: Report: NOT WRITTEN, run twice, the same fabrication signature both times. Disqualified. Real research works fine here, 13 sources fetched, 0/12 search failures, but FindingsWriter repeatedly fabricates citations to real sounding but never fetched Wikipedia URLs pulled from its own training knowledge, and PeerReviewer approves the rewrite anyway both times, only the separate grounding check catches it. Testing this candidate on vLLM ended in a reproducible fixed size OOM bug independent of context length, likely the multimodal vision encoder profiling pass, so we switched to Ollama, where nothink mode isn't honored (harmless here, since reasoning stays isolated and doesn't pollute content) but tool calling is clean. (Dagger, see the Qwen3 think mode note below.)

qwen3:4b plus a GRPO fine tune (thin_coverage). About 4.3GB (Q8_0 GGUF). Best result: roughly 1 to 2 out of 10 both times, still not_grounded, retry budget exhausted. Disqualified, though the targeted fix did work: there were zero thin_coverage stalls in either run, the exact failure the fine tune targeted is gone, confirmed 8/8 on held out eval too. It fails instead on a second, untouched failure mode: citation fabrication combined with real content getting dropped from synthesis. A structural fix (a grounding check warning was being silently truncated before it ever reached findings.md) measurably improved this on re test, grounded citations went from 0/8 to 3/9, but didn't fully close the gap: the model still sometimes cites a URL that its own source material explicitly flags as unverified when it has no real alternative. Only the training pipeline's own enable_thinking=False (applied directly via HF's chat template, no Ollama involved) is unaffected by the dagger caveat; the live Ollama benchmark run itself is not.

qwen3-4b-combined-v2-lora, a 7 dimension combined GRPO from 2026-07-28. Disqualified, cleanly. About 4.3GB (Q8_0 GGUF, merged and redeployed on 2026-08-19 as deepdelve-qwen3-4b-combined-v2). Held out eval went from 0.615 to 0.781, real generalization, not a ceiling effect, but two separate live benchmarks were both disqualified. The first live benchmark, on 2026-07-28, was confounded by the Qwen3 think mode passthrough Ollama bug, which burned 2 to 3 times more tokens on reasoning than intended and tripped an unrelated DeepDelve bug too, both root caused and fixed the same day. A clean re test on 2026-08-19 merged the still on disk LoRA fresh and deployed it via api.backend: "ollama" with enable_thinking: true (confirmed via a direct curl that think: true correctly isolates reasoning into its own field on this exact tag; think: false reproduces the bug, dumping raw chain of thought into .content). The result: a score of 0.25, worse than the confounded run's 0.5. final_report.md was the deterministic salvage banner, the model narrated in chat instead of ever calling write_workspace_file across its full writer retry budget, and findings.md repeatedly failed grounding on the same fabricated URLs (rentremote.com, nomadsembassy.com) across consecutive rebuild attempts. This confirms disqualification, not confounding this time: the fine tune's targeted objectives held, the held out gains are real, but citation fabrication and writer dispatch convergence, both untouched failure modes, are still broken at 4B scale even with reasoning cleanly isolated. That closes the "not yet done" clean re test caveat that had stood since 2026-07-28.

Ornith-1.0-9B (deepreinforce ai, dense, Qwen3.5 architecture, MIT license), disqualified cleanly on 2026-08-19. About 9.5GB (Q8_0 GGUF, deepdelve-ornith-9b-jsonfmt). Best result: a score of 0.000, with final_report.md as the deterministic salvage banner. This closes a long standing INCONCLUSIVE status: every prior run had an independent, non model confound attached, see parts 4 and 5 of the Literature Review for the full trail, a chat template empty <think> injection defect, a native backend tool call corruption bug, and two DeepDelve architecture bugs. All of those are fixed now. This re test used the fully patched deepdelve-ornith-9b-jsonfmt tag (0/18 web_search failures in its own prior live verification) via api.backend: "ollama" with enable_thinking: false, confirmed via a direct curl that think: false is actually the clean setting for this model on the native endpoint even with tools present, the opposite of what qwen3-4b-combined-v2-lora needed, so this really does have to be verified per model rather than assumed. The result: real research happened cleanly, 17 sources fetched, 22 findings, no tool call corruption, no infinite loop, but at the delegation quota exhaustion point, the Planner narrated its wrap up as chat prose instead of ever calling write_workspace_file, exhausting the full writer retry budget, and findings.md never got accepted at all. This is the same narrate instead of call failure class this project has now seen across the large majority of sub 14B candidates tried (Bonsai 8B, qwen2.5:3b-instruct, mistral:7b-instruct, mistral-nemo:12b, hermes3:8b, and this candidate too), and it's independently corroborated for this specific model family by two Reddit threads (part 4 of the Literature Review) as a harness agnostic trait, not something a serving layer or template fix reaches. This is genuinely the strongest research quality candidate tested at this size, matching its own earlier INCONCLUSIVE runs' cold start synthesis strength. The ceiling here isn't research capability, it's converting research into a written artifact under real task completion pressure.

Ministral-8B-Instruct-2410, disqualified at the schema stage on 2026-08-19. About 4.9GB (community tag nchapman/ministral-8b-instruct-2410:8b). Best result: fail, on an isolated smoke test, no full live run spent. This is a general purpose model, not a narrow function calling finetune, Apache 2.0/Mistral Research License, with a native long context of 128k. It isn't in Ollama's core library, so we used the community GGUF tag. The first smoke test, on the OpenAI compat endpoint with a simple single string arg web_search tool, failed even with tool_choice: "required", narrating a fake JSON schema as plain text. A native /api/chat endpoint smoke test with the same simple tool passed cleanly, real structured tool_calls, confirming an OpenAI compat translation layer issue for the simple case, the same class as this project's other Mistral family findings. But the real disqualifier was testing against DeepDelve's own actual delegate_tasks schema, an array of task objects rather than a single string: it failed 3 out of 3 times on the native endpoint too, narrating valid looking JSON as markdown text instead of a real tool call, every single time. The model can call a trivial single arg tool but breaks down specifically on the array of objects shape that is the Planner's first and most critical call in this project's whole pipeline. Disqualified at the smoke test stage per this project's own established practice, the same as llama3.2:3b and qwen2.5:3b-instruct, no full live run needed when the schema stage failure is this clear and reproducible.

Falcon3-10B-Instruct, disqualified at the smoke test stage on 2026-08-19. About 6.3GB (falcon3:10b, then bilel_cherif/falcon3-tools). Best result: 3 out of 9 real tool calls on an isolated smoke test, no full live run needed. TII's own model card confirms real function call training data (1.2 million posttraining samples including "function call data"), so this is genuinely a serving layer or packaging gap, not something assumed to be one. Ollama's core falcon3:10b tag flatly rejects tool requests ("does not support tools"), its TEMPLATE never references .Tools/.ToolCalls at all, confirmed via ollama show --modelfile, a packaging gap rather than a capability one. falcon3:7b isn't even a real Ollama tag. We found and pulled a community tag built specifically for this, bilel_cherif/falcon3-tools, "Falcon 3 10b for tool usage and function call," and its template does occasionally produce a real, correctly shaped tool_calls response matching this project's actual delegate_tasks array of objects schema exactly. But unreliably: 3 successes out of 9 identical isolated smoke test reps, about 33 percent, with the rest returning completely empty content and no tool call at all despite generating 77 to 82 tokens each time (something is generated and then silently dropped, not a clean template rejection). A pipeline that makes dozens of delegate_tasks shaped calls per run at roughly 33 percent per call success compounds to near certain full run failure, so this was disqualified on reliability grounds without spending a full live run, the same practice as the schema stage disqualifications above. This was re verified live too: we re pulled the tag and re ran all 9 reps a second time with raw, unedited curl output shown directly rather than summarized, after an earlier internal reporting mistake (noted below), and got the same result, 3/9 real tool_calls, one of those three malformed (tasks sent as a JSON string, not an array).

We actually researched the root cause of the silent empty response failure through GitHub rather than guessing. ollama/ollama#14958, "Tool calls silently drop with large system prompts," looked like a prompt length bug on the surface, but its real root cause, confirmed by the reporter after debugging with a maintainer, was a tool name mismatch: the model attempted to call a function name that didn't exactly match any registered tool (kebab case versus the real PascalCase name), and Ollama's server response for that mismatch is a silent empty response, content: "", no tool_calls, no error, even though the completion token count proves the model generated real output. It isn't specific to prompt length, that was just this particular reporter's own trigger shape. A second, still open issue, ollama/ollama#15539 on the gemma4 parser, shows the sibling failure mode from the other side: a validly generated tool call sometimes leaks as raw JSON text into content instead of getting extracted into tool_calls. Put together, Ollama's tool call extraction is fragile to any deviation between what the model actually generates and what the declared parser strictly expects, and its failure mode for that mismatch is silent, not an error. That directly explains both of Falcon3's observed failure shapes here, 6/9 silent empty and 1/9 leaked malformed JSON. It isn't that Falcon3 can't do tool calling, TII's own training data contradicts that, it's generation sample variance (temperature 0.8, non zero) meeting a parser too strict to tolerate it, with Ollama swallowing the mismatch instead of surfacing it.

Worth keeping on the record: midway through this investigation, an early observed "None" result after one earlier "True" result was reported to the user as "my extraction script had a bug," a cause stated before it was actually verified. That was wrong; the script was fine, and the underlying 3/9 result was real. It got corrected by re running live with raw, unedited output shown directly rather than summarized, so the finding didn't depend on trusting a prior claim. See the feedback_verify_before_stating_cause memory.

granite3.1-dense:8b and phi4-mini:3.8b. About 5.0GB and 2.5GB respectively. Both fail. Disqualified at the tool call smoke test itself: both narrate the call as literal text despite each model card explicitly claiming function calling support.

Phi-4 14B (Microsoft, MIT license), disqualified at the smoke test stage on 2026-08-19. About 9.1GB (jacob-ebey/phi4-tools). Best result: 0 out of 9 real tool calls on an isolated smoke test, no full live run needed. We researched this before pulling anything: Phi-4 (14B) and Phi-4-mini (3.8B, the candidate already disqualified above) are different models, not two sizes of the same one. Microsoft's own function calling support is documented for phi4-mini specifically; the 14B phi4 has no official tool calling support on Ollama at all, confirmed via ollama/ollama#9647, closed as "not planned" by the maintainers. The one community fix, jacob-ebey/phi4-tools, referenced directly in that same GitHub issue as "the only phi4 with tool calling support," was pulled and smoke tested against the real delegate_tasks schema, 9 reps, raw output shown live. The result was 0/9: every single response explained the correct JSON as prose ("Here's how you can format it: json {...}") instead of ever emitting a real tool_calls field. That's worse than Falcon3's intermittent 3/9, this is fully deterministic narration, not a parser extraction reliability problem. Either the community template doesn't actually wire tool call output at all, or the base 14B model, unlike phi4-mini, was simply never trained to invoke tools rather than describe them. Disqualified without a full live run, the same practice as the other schema and reliability stage disqualifications above.

llama3-groq-tool-use:8b. Best result: fail. Rejected at the tool call schema stage.

hermes3:8b. About 5GB 4 bit (vLLM, bitsandbytes). Best result: not_delegated, on two runs with different queries. Disqualified. It passed the isolated tool call smoke test cleanly, 3/3, a real structured array, no #6155 class bug, but under DeepDelve's real system prompt it narrates fake system text instead of ever calling a tool. The most severe case fabricated an entirely fictional "context length exceeded" error message and a fake retry narrative, confirmed invented, since vLLM's own server log shows no such error and KV cache usage was only 1 to 13 percent at the time, then repeated a similar fabrication pattern in a second, independent run on a different query.

mistral:7b-instruct. About 14GB bf16 cache, or roughly 5GB 4 bit at runtime (vLLM, bitsandbytes). Best result: not_delegated, on two runs with different queries, an identical failure both times. Disqualified, but the original "rejected at schema stage" reason is now wrong and superseded: retested on vLLM with settings.skip_chat_template_kwargs (a new fix that unblocks every Mistral family candidate from a permanent, by design vLLM restriction on Mistral tokenizer requests), the isolated tool call smoke test passed cleanly 3/3, a real structured array, no #6155 class bug. The real benchmark run reveals a genuine, different capability gap instead: the model consistently narrates its planned delegate_tasks call as literal markdown text instead of emitting a real tool call, even with --enable-auto-tool-choice, the same narrate instead of call failure class as Bonsai 8B and qwen2.5:3b-instruct, here happening at the Planner's very first dispatch.

MiniCPM5 1B, tried both as a single model replacement and as a paired specialist. The paired specialist run was confounded (a VRAM forced Planner/Builder swap off gpt-oss:20b), but the full single model replacement run was doubly corroborated: 0 delegate_tasks calls across two independent runs, Report: NOT WRITTEN both times. Disqualified in both forms tested, and the disqualification is final and doubly corroborated. The paired form is confounded per the project's own Model Evaluation Standard (point 2, isolate the candidate as the only variable) and we're not re litigating it per the user's own decision. The clean, isolated single model form reproduced the identical core failure on an independent re run, 63 events, zero real delegate_tasks calls. No further MiniCPM5 1B testing is planned.

Inconclusive, blocked, or not yet viable

These aren't disqualifications. Each one hit a real infrastructure or evaluation fairness blocker before the model's actual capability was ever cleanly tested, per Model Evaluation Standard point 1.

qwen2.5-coder:14b-instruct. About 9.9GiB weights (vLLM, bitsandbytes 4 bit). Result: inconclusive, not a pass and not a disqualification. An intermittent crash on first launch turned out to be transient rather than deterministic, a clean retry got past it. The first smoke test used the wrong parser (hermes), since this model was never trained on that convention, it uses <tools> tags and needs a community parser plugin (hanXen/vllm-qwen2.5-coder-tool-parser, reviewed and installed). With the correct parser, the isolated smoke test showed roughly 50 percent unreliable extraction, 2 out of 4 clean structured calls, and 2 out of 4 returned empty arguments: "{}" despite a normal completion token count. Real capability was never cleanly established either way.

MiniCPM3 4B, as a single model candidate. Result: blocked, not disqualified, and not re testable as is, a real infrastructure hang rather than a capability verdict. This candidate is genuinely promising on paper, with a documented BFCL v2 score of 71.6, an Apache 2.0 license, and native vLLM model support. A real hardware ceiling was found and correctly applied before benchmarking, roughly a 6144 token max feasible serving context on this GPU, under the project's 16K floor, but the run was allowed to finish anyway as informational. The result was a real hang, not a clean pass or fail: zero visible progress for over 16 minutes, traced to OpenBMB's own reference tool parser re scanning the entire accumulated generation text with a catastrophic backtracking risk regex on every streamed token, not anything DeepDelve's own code touches. It never reached the point of testing actual research or delegation behavior, so this doesn't count as a settled discard, it's an open infrastructure question (the reference parser needs incremental parsing, not full text re scanning) rather than "MiniCPM3 4B discarded."

MiniCPM4 MCP, as a specialist role candidate. Result: real infrastructure was built and kept, but the model itself isn't yet a stable candidate. The tool use SFT checkpoint's chat template emits a Python code block format (<|tool_call_start|>func(arg=val)<|tool_call_end|>), not OpenAI style JSON tool_calls, so Ollama's generic tool calling support fails outright against it. We built and kept finetune/minicpm_tool_proxy.py, a translation proxy verified against OpenBMB's own reference implementation after an initial pass missed two real gaps, anti repeat tool call guidance and keyword collision or hyphenated name argument parsing, both since fixed and reverified. With both fixes in place, the specific bug that motivated the proxy (task name as filename looping) did not recur, but the run surfaced different reliability problems in its place: search quota exhaustion from excessive re querying, a noisy search topical mismatch that the existing check correctly caught, and a real accuracy regression against an earlier run. Still not a stable specialist role candidate as of this evaluation, reusable proxy infrastructure but an inconclusive model verdict.

Tongyi-DeepResearch-30B-A3B (Alibaba-NLP/DeepResearch). About 18.6GB at Q4_K_M, or 13.5GB at IQ3_M. Result: two real benchmark attempts, both impractical. This is a 30B MoE model with 3.3B active parameters, trained specifically for long horizon research, but it's a single fine tuned model operating via ReAct/"IterResearch," not a multi agent system in DeepDelve's sense (no Planner delegating to typed specialists with independent context), and it has no published runtime grounding or citation verification layer comparable to DeepDelve's own. Chat template and tool call compatibility were confirmed clean, real structured tool_calls, not raw XML text, and a --depth quick trial confirmed real delegation behavior. At Q4_K_M, the run was killed at 1 hour 6 minutes, genuinely computing the whole time, real progress happening, just far too slow to be practical (this is also the run that exposed and got a real max_run_minutes bug fixed). At IQ3_M, it did worse on the real workload despite being faster on the trivial smoke test, 37 plus minutes against the actual Planner prompt with zero progress, no write_todos, no delegate_tasks at all. Not recommended for further local benchmarking without a materially different quant or a context and prompt length investigation into why the full system prompt specifically breaks it.

Not attempted

devstral:24b. About 47.1GB bf16, estimated around 17GB at 4 bit. Discarded on hardware grounds before any pull: the real weight footprint (confirmed via the HF API, since the repo's 94GB listing double counts two packagings of the same weights) is estimated at roughly 17GB even after 4 bit quantization, which exceeds the entire 17.1GB VRAM card before KV cache and overhead. Same standard applied to qwen3.6.

GLM 4.7 Flash and Ornith-1.0-35B. Ruled out on hardware grounds before any pull, their smallest available quants (19GB and 21.2GB) exceed this card's 17.1GB VRAM budget. No GPU time spent on either.

A candidate shortlist researched on 2026-08-19 via web search, compiled after qwen3-4b-combined-v2-lora's clean disqualification closed off the 4B fine tune path, looking for a lighter than gpt-oss:20b general purpose candidate, ranked by promise. Since then, Ornith-1.0-9B's clean re test finished on 2026-08-19 and was disqualified (see its own entry above). Ministral-8B-Instruct-2410 finished the same day and was disqualified at the schema stage (see its own entry above, no full live run needed). Falcon3-10B-Instruct finished the same day too and was disqualified at the smoke test stage (see its own entry above, no full live run needed). And Phi-4 14B also finished that day, disqualified at the smoke test stage (see its own entry above, no full live run needed). Deprioritized and not being re litigated: xLAM-2-8b-fc-r, watt-tool-8B, and ToolACE-8B, narrow function calling specialist finetunes on a Llama 3.1 8B base, already surfaced by a prior session's research pass (ROADMAP.md, 2026-07-18) and explicitly deprioritized then, since these represent "a real risk for the writer role per this project's own repeated lesson [narrow fine tunes overfit to schema correctness at the cost of general instruction following], not worth GPU time until a general purpose candidate looks more promising." That's still true; the four candidates above are all general purpose. There's also a literature lead not yet read in full, and per this project's own citation verification rule its findings shouldn't be cited as verified until it is: AgentFloor: How Far Up the Tool Use Ladder Can Small Open Weight Models Go? (arXiv:2605.00334). Its title is a direct match for this exact question, and it's worth a full read before committing further GPU cycles to this search, not just relying on this session's web search snippet.

Hosted

NVIDIA NIM, free tier. The best discovery quality of anything tried, but the free tier quota wall kills a multi agent run at around 10 minutes. It needs a paid endpoint, and this project is local only for now.

deepseek-v4-flash and deepseek-v4-pro (the DeepSeek hosted API). No local VRAM cost, it's a hosted API. Best result: 0.2/1.0 for flash (llama_judge, Colombia B2B complex item), and 0.0/1.0 for pro on the same item, both well below gpt-oss:20b's 0.7/1.0 baseline on the same rubric. Disqualified, both variants, as of 2026-08-04, across 4 total runs (two flash runs via the web UI/src/api.py, one flash headless CLI run, one pro headless CLI run), which clears the Model Evaluation Standard's 2 plus run bar for a discard claim. The root cause is genuine model unreliability, not infrastructure, and that was only confirmed after two real harness bugs found along the way got fixed first.

The first fix added api.backend: "openai_hosted" (in src/engine/orchestrator.py's _get_default_options/_HOSTED_PROVIDER_THINKING_EXTRA_BODY). The existing "openai" backend's thinking mode control (chat_template_kwargs plus reasoning_effort:"none") is a vLLM or local serving convention that DeepSeek's hosted API silently ignores, leaving thinking mode stuck at its own default, on, with effort set to "high," per DeepSeek's own thinking mode docs. This was confirmed via _agent_session.json, zero reasoning_content occurrences after the fix, that thinking really was disabled, but it didn't fix the failure, the outcome and timing were nearly identical before and after, so verbose narration as content, not hidden reasoning, turned out to be the real driver of the next bug.

The second fix addressed a check_task_verification_flagged guardrail starvation bug (in src/api.py's context_budget_chars cutoff, around lines 297 to 334). The old, unconditional force jump to the terminal or salvage branch gave this check zero real retry attempts before cutting the run off, since DeepSeek's narration volume alone blew the same 50,000 character budget that gpt-oss:20b never approaches. This was brought to parity with run_cli's existing two stage nudge then cutoff, one wrap up turn before the hard cutoff, not before.

With both of those fixed, the model still failed, and subagent_invocations confirmed the reason is structural to the model's own behavior, not the harness: DeepSeek re fabricates the same citations on redo. check_task_verification_flagged correctly never let the run advance to check_missing_findings (which is what dispatches the real FindingsWriter/Builder writer roles), because the same flagged task names kept recurring across retry attempts instead of resolving, a genuine reliability failure (repeatedly citing sources that don't match anything actually fetched, despite 32 to 51 real fetched sources sitting unused each run), not a config or budget artifact. deepseek-v4-pro actually scored worse than flash despite costing roughly 3 times as much, which rules out "just use the bigger model" as a fix.

It's worth being clear about what this candidate was NOT disqualified for: tool calling mechanics themselves were fine, zero tool call schema errors in 3 of 4 runs, and the 4th run's 37 tool errors were sub agents passing task names instead of filenames to read_workspace_file, a separate argument binding weakness worth naming but not the deciding factor here. It also wasn't disqualified over "doesn't have write access" confusion, since the Planner role genuinely has no write_workspace_file tool by design, DeepSeek's own statement to that effect was accurate, not a hallucination, it just never earned its way to the role that does have it.

Cross cutting notes

Every Qwen3 family row marked with a dagger above was very likely benchmarked with uncontrolled chain of thought reasoning, not the clean output its score implies. This was discovered on 2026-07-21 while auditing the same question for MiniCPM5 1B, confirmed live via a direct curl against Ollama 0.31.2 that neither chat_template_kwargs.enable_thinking: false (OpenAI compat) nor Ollama's own native think: false field suppresses Qwen3's reasoning. Native think: false is actually worse than doing nothing, since it dumps the raw unstructured chain of thought directly into message.content with no separate thinking field at all, while think: true correctly separates it out. Since DeepDelve's client (agent_framework) treats .content as the model's actual working output, every marked row was almost certainly reasoning polluted in its tool call arguments and written text throughout the whole benchmarked run, a real, previously unknown contributing factor to these disqualifications, on top of, not instead of, the capacity floor evidence in the References page. This was confirmed via a direct vLLM test to be Ollama's bug, not a Qwen3 model limitation: enable_thinking: false against Qwen/Qwen3-4B on a real vLLM server, a genuine chat template evaluation, gives a clean answer with zero <think> content and correct, unpolluted tool calling. One row above, qwen3-4b-combined-v2-lora, has now been re benchmarked through this clean path, see its own entry above; the rest of the rows above still haven't been, and their existing scores stand as the best available evidence, just not as clean evidence as previously assumed. The full trace is in the Changelog. One more thing discovered while doing that re test on 2026-08-19: "clean" via api.backend: "ollama" means enable_thinking: true, not false, since think: false on the native endpoint reproduces the identical bug (raw chain of thought dumped into .content, confirmed via a direct curl), and only think: true correctly isolates reasoning into its own field. The token budget cost this implies, reasoning always on, is a real, accepted trade off for a clean measurement, not a bug to work around.

There's also been a lower cost re test path available since 2026-07-28. At the time the think mode bug above was found, the only known fix was switching the whole serving stack to vLLM, a big step, since reverted (see the Changelog's "Ollama restored" entry). api.backend: "ollama" (ARCHITECTURE.md section 6) now gives the same clean nothink behavior directly through Ollama's native /api/chat endpoint, live verified for gpt-oss and Ornith-1.0-9B, no backend swap required. Re testing any dagger marked row through this path is now a real, low friction option, still not done, still an open call, but no longer blocked on a bigger infrastructure decision the way it was on 2026-07-21.

The meta result holds across every run and every model: no fabricated report has ever gotten past the grounding gates unlabeled. The defense layer is the validated product here; model quality only determines how often it has to fire. See the Changelog's bake off entry for the full trial history and untested candidates (Ministral-8B, two function calling specialist finetunes) noted for later.