Changelog Investigation Log 2

Changelog: Bake off Investigation Log II (2026-07-11 through 2026-07-21)

Part 4 of the Changelog, the later half of the model bake off and backend investigation log.

qwen3:8b vLLM re test, disqualified

Re run twice on fp8 dynamic quantization (bf16 alone left no room for KV cache). Both runs: final_report.md never written, retry budget exhausted on thin coverage, with the model repeating the identical narrated summary verbatim across completion check attempts, the same non convergence pattern already seen on qwen3:4b and this model's own earlier, think polluted Ollama verdict, now confirmed clean on a verified nothink backend. Closes off the "maybe it was Ollama's serving bug" possibility. A separate finding: the model repeatedly invented an invalid value for an optional MCP search parameter it should have left unset, fixed with an explicit prompt note. Fourth total occurrence of this non convergence class across models and backends.

InternScience/Agents-A1-4B, disqualified

vLLM was abandoned first for this candidate: three launches at different configs all failed at engine init trying to allocate exactly 256.00 GiB, a fixed size figure that rules out KV cache sizing, most likely a fixed size dummy input in the vision encoder profiling pass hitting a ROCm bug. Switched to Ollama with the official GGUF. Nothink mode wasn't honored by either disable mechanism, but reasoning stayed correctly isolated in its own field (not a correctness break, just cost). Two full benchmark runs both showed the same root cause: FindingsWriter fabricates citations to real sounding but unfetched Wikipedia URLs, regenerating the identical fabricated URL across independent rewrite attempts, and PeerReviewer approves the rewrite both times. The run correctly self terminated via the consecutive same problem escalation rather than burning the full retry budget, working as designed. Distinct failure mode from thin coverage non convergence, closer to a citation fabrication failure that resists correction even across multiple write/review/fix cycles.

MiniCPM5-1B, disqualified in both specialist and full replacement form

Tested as a paired Searcher/Analyzer specialist first. Integration was simple, Ollama's built in parser handled its XML tool call format natively. An isolated smoke test passed 5/5 with one gap (a delegation call dropping task instructions). A live end to end run produced the most favorable MiniCPM result seen: it correctly distinguished preprint, published, and self labeled "peer reviewed" sources. But this whole result turned out to be unintentional think mode, not nothink: Ollama doesn't evaluate the GGUF's embedded Jinja template and falls back to the Modelfile's simpler one, a documented, vendor acknowledged gap. Only vLLM and SGLang correctly implement the toggle.

A genuine nothink retest via vLLM told a worse story. The model committed early to a stale blog post as the "latest stable version," later fetched the correct current page, and still cited the correct page as the source for the wrong number anyway, traced with certainty to this model's own reasoning. findings.md shipped as an auto recovered draft, never actually passing the grounding check. Six identical malformed calls confused "fetch a URL" with "read a file already saved." Two placeholder author name strings reached the final report. Disqualifying regardless of mode. This verdict was later re flagged under the Model Evaluation Standard's isolation point, since Planner and Builder had been swapped off gpt-oss:20b to free VRAM; the placeholder names were attributable to that swapped Builder, but the source contradiction and malformed fetch calls were still cleanly MiniCPM5-1B's own. The user declined a clean paired retest, since gpt-oss:20b plus any small specialist is a strategy not worth trying again regardless of which small model fills the slot.

A clean single model run (MiniCPM5-1B in every role, the same bar every other candidate is measured under) settled it decisively: the model burned its entire think tool quota (~20 near identical calls) without ever calling delegate_tasks, then falsely asserted it had compiled findings from delegated tasks that never existed. The engine's own "not delegated" check caught it cleanly, no fabrication reached the user. Cleared every point of the Model Evaluation Standard, including a second independent run after ruling out contamination from an unrelated vLLM process hygiene bug found the same day; the second run reproduced the identical zero delegation failure. Final verdict: disqualified in both forms, doubly corroborated, no further testing planned.

Qwen3 family think mode confirmed broken on Ollama

Every Qwen3 benchmark row in the project's table was likely reasoning polluted. The standard OpenAI compat disable flag had zero effect on Ollama. Ollama's own native disable field was actively worse: with it set to disable, raw chain of thought got dumped straight into the content field with no separation, while setting it to enable correctly isolated reasoning into its own field. gpt-oss:20b is unaffected: its harmony format always produces an analysis channel by design, but Ollama keeps that channel cleanly separated in both cases, so its inability to fully suppress thinking is benign where Qwen3's is a real content pollution bug. A later structural scan of every run folder on disk confirmed no other disqualified candidate beyond the already flagged Qwen3 rows shows this signature.

The fix eventually landed without a backend swap: a new api.backend: "ollama" option (added 2026-07-28) reaches Ollama's native /api/chat endpoint directly and gives the same clean nothink behavior, live verified for gpt-oss and Ornith-1.0-9B. The bug was also confirmed Ollama specific via a direct vLLM test the same day: Qwen3-4B on vLLM (which actually evaluates the chat template) suppressed thinking correctly with zero leakage, confirming Ollama's failure to evaluate the embedded template is the whole gap, not a model limitation.

MiniCPM3-4B, left inconclusive

Promising on paper, documented BFCL v2 score of 71.6. OpenBMB's own repo (not the locally installed vLLM package's bundled parsers) ships a working parser and chat template; confirmed real structured tool calls after two small compatibility fixes. This model's KV cache cost forced a real serving ceiling of roughly 6144 tokens on this GPU, well under the project's usual 16K floor. Proportionally rescaling the context budget to fit was tried first and corrected by the user into a new standing rule: a candidate that can't clear ~16K tokens is discarded outright on hardware grounds, not accommodated. This run was let finish anyway since it was already informative. Result: a real hang, not a pass or fail, traced to OpenBMB's own reference tool parser re scanning the entire generation with a catastrophic backtracking regex on every streamed token, a parser bug, not a DeepDelve one. Killed rather than let it burn GPU time. Never reached testing actual research behavior, so it stays an open infrastructure question, not a settled discard.

MiniCPM4-MCP, specialist role candidate, infrastructure kept, model not viable

This model emits a Python code block tool call format instead of OpenAI JSON. A FastAPI translation proxy was built, tailored to the model's documented format, translating both directions and verified correct in isolation. Config plumbing added a specialist base URL override alongside the existing specialist model option; a real bug (the HTTP client still hardcoded to the main endpoint) was caught and fixed during wiring.

A first live run completed with a real report and found one real academic source the baseline gpt-oss run had missed, but surfaced a new failure: a nested Analyzer sub agent called file reading tools with its own task label instead of a filename, repeatedly, requiring 10 fresh redispatches, and reliability was meaningfully worse under real load (53 tool errors versus 0-8 baseline). The proxy infrastructure was kept as reusable; the model itself wasn't yet a viable specialist candidate.

A correction followed the same day, caught directly by the user ("I told you to search implementations... if it's new we need to read documentation"): the verdict had been reached before reading OpenBMB's own reference implementation, which turned up two real gaps, missing anti repeat tool call guidance and a simpler parser that silently dropped hyphenated/keyword colliding argument names. Both fixed. A third run with the fixes confirmed the original bug (task name as filename looping) was gone, but different reliability problems surfaced in its place: quota exhaustion from excessive re querying, and a final report that regressed in accuracy, citing a blog aggregator instead of the real paper. Verdict: the targeted fix worked, but the model's reliability in a real multi step role stayed inconsistent run to run, still not a stable specialist candidate.

Tongyi-DeepResearch-30B-A3B, two attempts, both impractical

A 30B MoE (3.3B active) model trained specifically for long horizon research, but architecturally a single fine tuned model, not a multi agent system, with no runtime grounding layer comparable to DeepDelve's own. A community GGUF's chat template correctly parses its native tool call XML, confirmed live, and a quick depth trial showed real delegation with real fetches. Two full benchmark attempts followed. At Q4_K_M, the run made genuine progress (real delegation, real fetches) but was killed at 1h6m by a max_run_minutes bug this exposed and fixed, simply too slow to be practical. A lighter IQ3_M quant, expected to be faster, was worse on the real workload: 37+ minutes with zero delegation and no todos written at all, despite passing the trivial smoke test easily. Neither quant passed a real trial against the full multi thousand token Planner prompt. Not recommended for further local benchmarking without a different quant or a context length investigation.