Changelog Investigation Log

Changelog: Bake off Investigation Log I (2026-07-11 through 2026-07-21)

Part 3 of the Changelog, the earlier half of the model bake off and backend investigation log. Full evidence trail behind Model Bake off's verdict summary.

Early Ollama candidates (2026-07-13 to 2026-07-18)

granite3.1-dense:8b and phi4-mini:3.8b both failed the delegation smoke test outright, narrating the tool call as literal text every attempt despite each model card claiming function calling support, same disqualifying class as devstral:24b. gemma-4-12b and bonsai-8b both passed the smoke test. Gemma's full benchmark came back Report: NOT WRITTEN after 33 minutes, but a clean, honest failure that validated that day's 5 reliability fixes end to end (26/26 search calls, 27 sources, 4 ungrounded findings correctly rejected); the model itself hit 22 rejected delegation calls and a late reasoning loop. Bonsai's benchmark failed worse: research worked fine (22 findings, 15 sources), but every FindingsWriter attempt across all 8 tries "finished" with a bare empty text response, never calling the write tool, and PeerReviewer said "REVIEW: CLEAN" without ever reading the file. This exposed a real gap (a clean verdict wasn't cross checked against actual read calls), fixed the same day by checking the read quota's used count delta around the reviewer dispatch.

qwen3:4b looked like a stronger Bonsai replacement (2.5GB, 4x Bonsai's context, established tool track record) and passed delegation cleanly with real semantic task routing. But it showed a third distinct non convergence flavor: as FindingsWriter on a trivial query it called the write tool 10 times with near identical content, only the write quota stopped it. Its full benchmark came back Report: NOT WRITTEN after retry budget exhaustion on thin coverage: only 1 real source landed against 4 tasks, and all 8 retries got the identical canned non response instead of adapting. Disqualified. qwen3:8b hit the same disqualifying shape (Report: NOT WRITTEN, thin coverage, 2/6 tasks sourced) but with a distinct failure signature: instead of dispatching a writer, the Planner narrated the full report inline as chat prose with a "Stop here." sign off, and nothing existed on disk.

llama3.2:3b failed the schema stage: the tasks array parameter came back as a JSON encoded string 3 of 3 times. Root caused, not assumed: feeding the model its own real validation error made it abandon structured calling entirely rather than fix the array. Confirmed as a known, unresolved upstream Ollama issue (ollama/ollama#6155, open since August 2024), then confirmed model side, not serving side, via a controlled 3-backend test: llama.cpp (own official ROCm build) and native vLLM both reproduced the identical stringification bug, while qwen2.5:3b-instruct passed cleanly on the same servers. Same model, same failure, three unrelated tool call extraction mechanisms; definitively the model, not any backend. qwen2.5:3b-instruct itself passed the schema stage but failed its full benchmark on the writer tier, the same "finishes without ever calling write" root cause as Bonsai, now confirmed twice that the read-quota-delta hardening catches it correctly.

gpt-oss:20b was re confirmed live the same session: a real grounded report in 18 minutes, 15 sources, 0 search failures, the only one of four same day attempts to produce a written report at all. Formally scored on 2026-07-18 against the benchmark rubric: 6/10, "usable with manual verification." Full marks on structural integrity, heuristic coverage, and quantitative grounding; zero on architecture coverage (a real but topically different literature set, not a fabrication) and zero on Colombia cultural context, despite a Colombia holidays page having been fetched cleanly, a second independent confirmation of the shared quota pool starvation bug logged elsewhere.

Heterogeneous role tiering: implementation and A/B test

Tiered dispatch (gpt-oss:20b for roles needing self correction, an optional settings.specialist_model for leaf roles) was implemented cleanly with no schema changes. A VRAM probe run before any code confirmed the card can't hold two models resident at once, meaning gpt-oss:20b never actually unloads between specialist calls, so tiering only adds a second model competing for the same fixed budget rather than reducing peak pressure. The probe's own numbers already predicted the outcome; a later 4.2x slowdown (4513s vs. 1079s baseline) was just the confirming measurement, and should have been a go/no-go gate before running the A/B test at all, not a footnote alongside it. Worse, the run's own content silently dropped the entire main query topic despite the specialist model having fetched genuinely relevant sources for it, a new instance of the project's recurring "real content vanishes at synthesis" pattern with no visible quota exhaustion trigger this time. Verdict: the tiering code is correct and reusable, but this pairing on this hardware was a net loss. specialist_model left unset in the live config.

vLLM bake off re test (started 2026-07-21)

Motivated by two confirmed Ollama serving bugs (Qwen3 think mode passthrough, nested array stringification) that could mean some Ollama-era disqualifications reflect a serving bug, not real model incapability. Two pre flight checks cleared first: needed HF repo IDs existed, and bitsandbytes on ROCm passed a real functional 4-bit quantization test, not just an import check. One operational lesson from the same spike, applying to every vLLM kill from then on: a hard kill signal orphans the engine child process and leaks VRAM, always terminate first and only escalate if memory is still held.

Candidate	vLLM re-test result
mistral-nemo:12b	Blocked, not disqualified. Tool call smoke test passed cleanly (Ollama bug confirmed absent), but the real run failed 100% of the time on a 400 error: vLLM's Mistral tokenizer rejects any request with chat template kwargs, which DeepDelve's client always sends. A same day fix (settings.skip_chat_template_kwargs) unblocked this and two other Mistral family candidates, but this one wasn't yet re-run with the fix.
llama3-groq-tool-use:8b	Disqualified on real evidence. First smoke test was invalid (its chat template has no tool rendering logic at all). Rebuilt the documented raw prompt format by hand: JSON generation was genuinely good (3/3 well formed, real nested array, Ollama bug absent), but it consistently omitted the required XML wrapper tags every time, confirmed mandatory by both the upstream reference parser and vLLM's own bundled parser. A caught mid-investigation misstep: an early attempted fix (priming the assistant turn with a literal opening tag) was an unsourced generic trick, not checked against the model's own docs first; the user stopped it and asked directly whether docs had been consulted. They hadn't. Going back to primary sources produced the real, dual confirmed verdict above.
qwen3:8b	Killed mid run, no verdict, after a real DeepDelve-side bug surfaced. The run was progressing well (18 fetched URLs, 19 findings) when a user requested cross check against the sources folder found only 15 files on disk and 5 of 19 findings citing a fabricated URL, a leaked task name string. Root caused as model agnostic: the dispatch loop's fallback rendered a task name as a source URL whenever a task fetched nothing of its own, with no marker distinguishing it from a real citation. Fixed by recovering the real reference URL from the Searcher-to-Analyzer handoff before falling back, and by moving any remaining non URL fallback into an explicitly non-citable list. No prior verdict needed reopening; qwen3:8b was the natural next candidate to retest.