Changelog Recent 2

Changelog: Recent, Part II (2026-07-26)

Part 2 of Recent in the Changelog, continuing the same busy day, 2026-07-26.

v2 combined LoRA disqualified, twice

First live benchmark: final_report.md never written, retry budget exhausted, citations flagged as fabricated twice against the same 3 tasks, and the model narrated report content in prose instead of calling the write tool. Confirmed both this candidate and base qwen3:4b ran under the already known Qwen3 think mode passthrough bug, a pre existing confound that doesn't change the verdict since base qwen3:4b carries it too and still failed. Cleanup followed the disqualified candidate hygiene rule: tag and intermediates removed (~11.6GB reclaimed), adapter kept as the training artifact.

A clean re test through the native Ollama backend on 2026-08-19 confirmed think: true correctly isolates reasoning there. One process mistake happened first: an edited live config was never actually read by the eval harness (it always builds its own from the template unless a config path is passed explicitly), wasting a ~47 minute run that silently re benchmarked the baseline instead of the candidate. Corrected by passing a config path explicitly and verifying the materialized config. Result: score 0.25, worse than the earlier confounded run's 0.5. final_report.md was the deterministic salvage banner, findings.md repeatedly failed grounding on the same fabricated URLs. Confirmed: the fine tune's targeted gains hold, but citation fabrication and writer convergence, dimensions the reward never targeted, are still broken at 4B even with reasoning cleanly isolated. A real capability ceiling, not a serving artifact.

The same day, a deeper root cause was found for why retries never resolved it: the Qwen3 think passthrough bug inflated every turn's token count 2-3x, blowing the context budget in just 2 of 8 completion check attempts. The TUI's forced-final path then swapped the real, still retriable problem (citation fabrication) for a low severity hygiene check that explicitly promises never to block a run from finishing, so that check got reported as the terminal blocker instead. Fixed with a parameter marking that check as never allowed to become the final blocker once a cycle is already headed to the final branch. The disqualified verdict above stands as recorded but is now understood to be confounded, the model never got its real 8 attempt budget; a truly clean re benchmark wasn't done this session, by explicit scope, fix the bug, don't re benchmark yet.

Ollama restored as the permanent serving backend, vLLM removed

Reverses an earlier move to vLLM, after the same day's vLLM re test sweep disqualified or discarded 9 candidates in a row, several with serving layer shaped symptoms (silent zombie crashes, intermittent empty JSON, a garbled tool call marker). Primary sourced research confirmed bitsandbytes on ROCm is real but young on consumer RDNA, this hardware ran it at its minimum required version floor, while llama.cpp's HIP backend (what Ollama uses) turned up zero documented quantization correctness issues. Most of that day's actual disqualifications weren't ROCm caused though, the same non convergence and narrate instead of call patterns reproduced identically on Ollama too. This is an explicit tradeoff to fix the flaky crashy tail, accepting the two known, still real Ollama bugs (Qwen3 think passthrough, nested parameters) in exchange for llama.cpp/HIP's serving maturity edge, not a claim Ollama is bug free. The vLLM venv and its HF cache checkpoints were deleted; DeepDelve's own shared cache dependencies were preserved.

Mistral family vLLM re tests

mistral-nemo:12b re tested with a chat template fix: 6/8 clean on the isolated smoke test, real engagement in the full benchmark (13 sources, 0/7 search failures, 8 findings), but ended with retry budget exhausted on thin coverage, the same non convergence signature documented 3 times before across models and backends. Disqualified, joining the thin coverage family rather than the narrate instead of call one.

devstral:24b discarded on hardware grounds without a pull: the HF repo's 94.3GB figure double counts two packagings of the same ~47.1GB weights, which would still land around 17GB quantized, exceeding the entire 17.1GB card before KV cache.

qwen2.5-coder:14b-instruct went through three rounds before an honest verdict, each correction caught by direct user pushback on an over confident claim. Round one misdiagnosed a silent zombie crash as a system RAM issue (a misread log warning about network filesystem prefetch, irrelevant to this local mount); discarded on hardware margin grounds instead. Round two: the crash didn't reproduce on retry, so round one was wrong on its own terms; the isolated smoke test found 3/3 consistent failures, well formed tool calls wrapped in the wrong XML tags for the parser in use. Round three: confirmed via primary sources this variant was never trained on the Hermes convention; installed the correct community parser after reviewing it directly. Result: 2/4 consistent successes, 2/4 empty arguments despite normal token counts, not disambiguated whether model or parser fault. Verdict: inconclusive, not disqualified, real capability was never cleanly established, a full benchmark was never attempted.

hermes3:8b passed its isolated smoke test cleanly (no thinking mode in this template at all) but in the real benchmark fabricated an entirely fictional context length error as narrated text instead of calling any tool, confirmed invented by grepping the serving log (no such error existed, GPU KV cache usage was only 1-13%). A second run with a different query reproduced the same not_delegated root cause with a different hallucinated excuse. Disqualified, a more severe narrate-instead-of-call variant than seen before, fabricating entirely fictional error text.

A config fix (skip_chat_template_kwargs, a new explicit option rather than model family auto-detection) unblocked all three Mistral family vLLM candidates at once, root caused directly at vLLM's source: the Mistral tokenizer permanently rejects any chat template keyword arguments, not a version-fixable bug. With the fix, mistral:7b-instruct got past the 400 error into a genuine capability failure: not_delegated on two independent runs, the model narrates its delegation call as a markdown code block instead of a real tool call even under explicit tool choice. Disqualified, same narrate-instead-of-call class, but hitting at the very first Planner call rather than the writer stage.

qwen2.5:3b-instruct disqualified on vLLM with a new failure mode: the model reliably emits a malformed tool call JSON missing the outer closing brace, verified as a genuine model defect (not a parser bug) by testing the parser's extraction directly against the captured response text. meta-llama/Llama-3.2-3B-Instruct also disqualified on vLLM, and this result mattered beyond the one candidate: it reproduced the identical stringified-array bug (4/4) on a completely different serving stack than Ollama, directly disproving the hypothesis that this was an Ollama-only serving artifact for this candidate, confirmed by vLLM's own "Known issues" docs attributing it to the model's own generation behavior.

qwen3.6 (35b a3b) was discarded on hardware grounds without testing: its FP8 checkpoint alone is 37.5GB, over double the VRAM budget, no 4-bit prequant exists. Status after this session: gpt-oss:20b itself deliberately deprioritized to last as the lowest information value re-confirmation remaining.

Three more production bugs, plus a Gemma 4 fine tune disqualified

Each caught by actually reading final_report.md after a smoke test, not just confirming a completion check fired. First, several grounding checks compared URLs with plain string equality; a percent-encoded citation didn't match a raw Unicode fetched URL, false-flagging a correct citation and producing an empty report despite 27 real sources. Fixed with a URL normalizer applied at every comparison site.

Second, delegate_tasks shares one global quota across every tier with no per-task sub budget; a smoke test showed one WebSearcher task calling it 6+ times in a row despite its own prompt saying to stop early, consuming most of the run's budget on one facet. A specialist delegation cap (default 3) was added. A bug in the fix itself got caught before commit: the first version used a plain contextvar that silently never accumulated across separate async tool call tasks; fixed by mirroring an existing mutable list pattern.

Third, the task verification check had no notion that a flagged task could be superseded when the Planner renames it on retry; a live run renamed a flagged task 3 times, two renames actually succeeded with real verified findings, but the check kept nudging the stale original names, burning the entire retry budget with zero report. Fixed by downgrading a flagged entry to "superseded" when its instructions closely match an already verified task.

Fourth, found while running a Model Evaluation Standard raw API check on the Gemma candidate below: the default options builder now also sends an explicit reasoning effort of "none" alongside the thinking disable flag, since some models ignore the flag but respect the effort setting over Ollama's OpenAI compat endpoint.

yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF (distinct from the other tracked Gemma 4 12B row) was disqualified, run twice, identical signature both times: the model calls write_todos with byte-for-byte identical arguments repeatedly, ignoring an explicit tool error telling it to stop, until DeepDelve's own anti-loop safety net force-aborts the run. Research itself worked fine both runs, real sources fetched, zero search failures; the model simply never transitions to writing the report. The most severe agentic reliability failure documented in the bake off to that point, and not a serving layer issue, since nothink and tool calling were independently clean at the raw API level.

Planner redelegation loop fix and two more production bugs

Root caused why the Planner renames the same research angle across retries instead of redispatching under the same name (recurring as 43 calls for one angle in a resumed run): two checks' own wording told it to "redelegate... phrased differently," which the model conflated with inventing a new task name. Fixed the wording and added a non-blocking similarity nudge in delegate_tasks flagging likely renames; live confirmed it reduces but doesn't eliminate the pattern.

A live smoke test hit a worse empty-response case than an earlier fix assumed: FindingsWriter produced nothing usable across six consecutive completion check attempts despite 61 real findings existing the whole time. Added a deterministic, non-LLM salvage path that writes findings.md directly from already-assembled evidence when both writer attempts fail. A second bug found while verifying this fix: the URL extractor didn't stop at a trailing backtick, breaking every inline-code style citation, which would have defeated the new fallback on exactly the content it exists to rescue.

Caught by reading a live final_report.md, not just confirming checks fired: a balanced two-facet query produced a report 100% about one facet, the other vanished entirely at FindingsWriter consolidation and nothing caught it, since coverage looked balanced and the underuse check compares against findings.md, which had already lost the data. A new check compares findings.md's cited URLs against the real per-task research record and fires when a covered task has zero of its real URLs represented. A sixth GRPO reward dimension was added mirroring this check, inert prep only, not yet wired into training.

A later smoke test confirmed the deterministic salvage firing organically for the first time live, but the run still failed: FindingsWriter cited a fabricated near-duplicate URL across 7 dispatches despite an embedded grounding warning, the same recurring shape logged two days earlier in a different domain. Researched properly before fixing (negation is documented as fragile, naming forbidden content can prime its reproduction; CRAG and Self-RAG both structurally filter flagged evidence rather than annotate it). The shared citability predicate now excludes findings carrying a verification warning marker, reversing an earlier deliberate choice not to exclude them. Since the same evidence string feeds both FindingsWriter and the salvage fallback, this one change fixed both the recurring fabrication and the "salvage inherits poisoned evidence" problem at once. Replayed against the real failing run's data to confirm. A fresh run the same day confirmed no recurrence, FindingsWriter converged normally.

Later still, re-checking two deferred research ideas: one stayed deferred as a real future direction. The other was based on an incorrect assumption, caught by re-reading the actual dispatch code: full grounding (including NLI contradiction detection) already ran for Searcher dispatches. The real gap was that Analyzer leaf dispatches never got it at all, only a narrow reconstructed URL check, so Analyzer findings (the shape most likely to misreport a figure from a single document) had materially weaker coverage for no principled reason. Fixed by adding the same proven Searcher-tier grounding call to the Analyzer branch alongside the existing check. Live confirmed the same day: two Analyzer findings got flagged with problem types that were structurally impossible before the change.

VERIMAP Phase 1: a structural per-task verification ledger

Shipped the same day. The literature review's VERIMAP paper has a planner author an explicit verification function per subtask, executed by a separate verifier before a coordinator proceeds. DeepDelve's own check pipeline produces exactly one verdict per attempt for the whole run, no per-task pass/fail record existed, and several bugs fixed this session (starvation guard, uneven investment, findings underuse) were individually patched symptoms of that same missing dimension.

A design tension got resolved before building: the paper has the Planner author each verification function, which conflicts with this project's own lesson that local models are unreliable at following new structured conventions. Reconciled by having the engine compute the ledger structurally from existing ground truth instead, keeping the paper's real contribution (a task has its own checkable verification state) while dropping the part that doesn't fit.

A new per-task ledger, keyed by task name, gets recomputed fresh every completion check attempt from findings and the citability predicate, purely additive, no new Planner-facing field or prompt convention. A new completion check, the first genuinely task-scoped one, fires when every finding for a task was excluded by the citability predicate, naming that specific task rather than nudging the whole run generically. Explicitly deferred: actually independent per-task redispatch that bypasses the Planner's own turn, real separate scoping work, worth doing only after seeing how often the new check recurs on real runs.