Changelog

Changelog

Dated history of the DeepDelve model bake-off, live-testing incidents, and architecture fixes. Moved out of ROADMAP.md to keep that file focused on what's currently open.

History

2026-08-17 (latest): Seven real bugs found and fixed across 6 live runs, closing the "per-source evidence-crowding" item that stood as the #1 priority since 2026-07-22

Closed out every open item from session_status/2026-08-16.md, then kept live-testing the same "Lisbon vs Mexico City" prompt through 4 more full runs, root-causing each new failure via direct _run_state.json/session-transcript reads rather than trusting log lines alone. Full traces in ARCHITECTURE.md §2/§3's updated landmine writeups; summary here, roughly in the order fixed:

1. RunState.coverage() empty-summary exclusion (commit 7eefe5e) — a real, http-prefixed source_url used to count as "covered" even when its summary was completely empty (a sub-agent ending its own turn immediately after a tool call, zero trailing text, no cutoff marker at all — a third, previously-undetected synthesis-vanishing mechanism, measured at 25%/42% of all findings across two live runs). coverage() now excludes via the same _is_null_finding_summary predicate used elsewhere, fixing check_thin_coverage/check_uneven_task_investment automatically (both consume coverage()'s output directly).
2. Ledger rollup + FOLLOW-UP DIRECTIONS grounding fix (commit 87b2d73) — a depth==1 task whose OWN findings were all empty still had real evidence in its depth>1 Analyzer children that _update_task_verification never credited back to it; new top_level_task_name contextvar fixes the rollup. Separately, a specialist's FOLLOW-UP DIRECTIONS: section (suggested next URLs, never a citation) was firing false SYSTEM VERIFICATION WARNINGs that invalidated genuinely-cited real content next to it.
3. Task-name-churn denominator inflation fix (commit 182d2e6) — _looks_like_renamed_task's raw difflib char-ratio scored 0.11 on a real live full-sentence paraphrase (nowhere near its 0.6 threshold), so a facet redispatched 3 times under 3 different names was never recognized as one rename — and even when caught, coverage() never read the superseded ledger status, so the denominator kept inflating anyway. Fixed both: a content-word-overlap OR-trigger, and coverage() excluding superseded task_names.
4. URL-scoped verification-warning exclusion (commit 4c5a4c3) — root cause of the "per-source evidence-crowding" item first named 2026-07-22 and repeatedly flagged since as the clear #1 priority (session_status/2026-08-16.md item 3): add_finding attaches ONE shared synthesis text to every URL fetched in a turn, so a stub_source/unverified_urls flag about ONE co-fetched URL wholesale-excluded the record for ALL of them — confirmed live, a real MX$17,300/month price got thrown away alongside an unrelated stub-page flag. _is_citable_finding now scopes the exclusion to the finding's own source_url when the marker names specific bad URL(s).
5. read_workspace_file exact-repeat quota dedup (commit b442797) — a FindingsWriter dispatch called read_workspace_file with IDENTICAL arguments 2-3 times in a row, burning its entire quota before finishing its actual edit work (confirmed: "41/47" quota-exhaustion, forcing a final_report.md-never-written outcome). Live-confirmed working: zero quota-exhaustion events in the very next live run, versus the prior run's clear 41/47.
6. Warning-marker leak fix (commit ba8dddf) — a direct side effect of fix 4, found in the NEXT live run after it shipped: a finding correctly staying citable (its own URL wasn't the flagged one) still carried the raw warning marker TEXT verbatim in its summary, which got rendered into findings.md by the deterministic fallback and then re-flagged by findings.md's own grounding check as if the warning's named URL were a real citation — a self-inflicted loop, confirmed via 3 byte-identical findings.md.rejected_attempt_N snapshots. Fixed by stripping the marker text before rendering a citable finding's block.

Two real issues found but deliberately left open, not rushed: (a) a writer-role dispatch (FindingsWriter) can end its own turn with zero output after a writer_gate_ctx block — the same "zero trailing text" mechanism as fix 1, now confirmed to also hit a writer role, where the consequence is worse (nothing written at all); the existing one-shot retry only partially absorbs it. (b) why FindingsWriter's rebuild sometimes fails to self-correct a flagged citation across multiple retries. Both need their own properly-scoped investigation, not a tail-end patch — see session_status/CURRENT.md for the live-run evidence backing each.

2026-08-16: Four real completion-check bugs found and fixed by repeatedly live-testing one prompt instead of trusting a single pass

Ran the medium "Lisbon vs Mexico City" eval prompt (eval/dataset.jsonl) as a smoke test for an unrelated open item (_find_sibling_fetch), and instead of getting a clean confirmation, kept hitting real, DIFFERENT bugs each of 5 consecutive live runs (~45-68 min each). Chased each one down to its actual root cause via _run_state.json/findings.md/final_report.md (not log lines alone — a mid-session user correction, "you're too confident," caught a case where the run's "System" log messages looked like healthy progress while the actual report on disk was still badly incomplete) rather than retrying blind. Full root-cause traces in session_status/2026-08-16.md and ARCHITECTURE.md §1/§2's own updated landmine writeups; summary here:

1. gap_acknowledged directive-oscillation fix (commit cdf354c) — check_task_verification_flagged's quota-exhausted "stop redelegating" directive could get silently reversed by a LATER retry_quota_topup refill flipping quota_exhausted back to False, reissuing "redelegate" for a task already told to stop. gap_acknowledged on the per-task ledger entry now makes that decision sticky across the ledger's full recompute.
2. Cross-tier completion-check starvation fix (commit 1e70bba) — the existing starvation guard only protected GROUNDING_CHECKS when the SAME COMPLETION_CHECKS problem repeated consecutively; a run where a DIFFERENT COMPLETION_CHECKS problem won every attempt starved GROUNDING_CHECKS (specifically report_underuses_evidence, built to catch exactly this) for an entire run despite final_report.md having dropped 3 of 4 requested facets. New _COMPLETION_TIER_PROBLEMS/_consecutive_tier_wins generalize the guard to the whole tier. Live-confirmed working: post-fix, the same check correctly kept firing and dispatching Builder across 11 attempts instead of never getting a turn.
3. writer_gate_ctx / edit_workspace_file fix (commit 1e70bba) — the structural gate forcing FindingsWriter to write before reading only accepted write_workspace_file, but is armed for the per-facet ADD-ONLY correction dispatches too, whose own instructions say to use edit_workspace_file instead and never touch the rest of the file. The gate's block (wording: "call write_workspace_file now") actively steered the model toward exactly the full-file overwrite its own instructions forbade, silently destroying facets a prior round had already added. Fired 20 times in one run that never converged. Live-confirmed working: zero occurrences post-fix, findings.md converged cleanly on the first pass.
4. _looks_like_renamed_task entity-mismatch fix (commit 1e70bba) — the heuristic meant to catch the Planner renaming a flagged task instead of retrying it under the same name false-positived on two INDEPENDENTLY dispatched, differently-named tasks sharing a template (two cities' rent facets differing only in city/neighborhood names, 0.89 raw difflib similarity). This silently, permanently marked one city's real facet "superseded" the moment the other verified — it never appeared in findings.md or the final report, with no gap ever flagged anywhere in the run. Fixed via proper-noun extraction + a Jaccard-overlap override that only trusts a high text-similarity ratio when the two tasks' actual named subjects overlap. Found too late in the session for a fresh live run to finish before this write-up — unit-tested, live reconfirmation still open (see session_status/CURRENT.md).

Related, found but deliberately NOT fixed this session (own future investigation): a single task that legitimately fetches many real sources (8+ seen live) can still crowd FindingsWriter's one-shot synthesis even after the existing per-facet dispatch fix (which scopes by TASK, not by individual source) — confirmed live, 11 of 13 source headings in one findings.md got a placeholder despite real content existing for at least one of them, and Builder then fabricated plausible numbers for those gapped sources in the final report. See session_status/CURRENT.md item 2.

2026-08-04: First hosted frontier-model test (deepseek-v4-flash/deepseek-v4-pro) — DISQUALIFIED, but two real cross-cutting harness bugs found and fixed along the way

First-ever hosted (non-local) model tested against this project's real pipeline, via src/api.py + web UI and the headless eval/evaluate.py harness. 4 total runs (2x flash via web UI, 1x flash headless CLI, 1x pro headless CLI) against eval/dataset.jsonl's complex-difficulty Colombia B2B item — DeepSeek's own thinking-mode/tool-calling docs (api-docs.deepseek.com/guides/thinking_mode, /guides/tool_calls) were read directly before any config change, not assumed from memory, after an initial wrong guess at DeepSeek's current model names (deepseek-chat/deepseek-reasoner, both stale — current models are deepseek-v4-flash/deepseek-v4-pro) was caught and corrected via the real pricing docs first.

Two real, reusable harness bugs found and fixed, both worth more than this one model's own verdict:

1. api.backend: "openai_hosted" (src/engine/orchestrator.py's _get_default_options + new _HOSTED_PROVIDER_THINKING_EXTRA_BODY provider map, src/config.py's new OPENAI_API_BACKEND env overlay): the existing "openai" backend's thinking-mode control (chat_template_kwargs + reasoning_effort:"none") is a vLLM/local-serving convention a real hosted API just silently ignores — confirmed live that DeepSeek defaulted to thinking ON at effort high regardless, and that chat_template_kwargs is meaningless to its API at all. "openai_hosted" looks up each hosted provider's own documented convention instead, keyed by openai_base_url. Confirmed fixed via _agent_session.json showing zero reasoning_content occurrences post-change — but this alone did NOT fix DeepSeek's failure, isolating that its narration is verbose-as-content, not leaked hidden reasoning.
2. check_task_verification_flagged guardrail-starvation bug in src/api.py (its context_budget_chars cutoff, was an unconditional run_state.attempt = 10**6 force-jump): confirmed live that a verbose model can blow the same 50000-char budget gpt-oss:20b never approaches, before the run's FIRST completion-check attempt even happens — giving a check that was actively working (verified-task count improving run-over-run) literally zero real retries before salvage. api.py's own code comment had explicitly, deliberately chosen this blunter behavior over run_cli's existing two-stage nudge-then-cutoff for shared-queue-protection reasons; brought to parity anyway since one bounded extra wrap-up turn doesn't meaningfully weaken that protection and the alternative (a guardrail that can never fire) is worse. Verified the fix works: a subsequent run's completion_check_attempts showed real numbered attempts (0,1,2,3) with task_verification_flagged correctly escalating and yielding to missing_findings per its own documented 3-strike design, instead of jumping straight to the 1_000_000 sentinel with zero attempts like both pre-fix runs did.

With both fixed, DeepSeek still failed — root cause is genuine model unreliability. Score: 0.2/1.0 (flash), 0.0/1.0 (pro, worse despite ~3x the price) vs. gpt-oss:20b's 0.7/1.0 baseline, same rubric. The real, confirmed cause: DeepSeek re-fabricates the same citations on redo — check_task_verification_flagged correctly refused to let the run advance to the check that dispatches the real FindingsWriter/Builder writer roles, because the SAME flagged task names recurred across retry attempts instead of resolving (confirmed via subagent_invocations: zero Builder/FindingsWriter dispatches in the run that reached this state). Not disqualified for: tool-calling mechanics (clean in 3/4 runs) or "no write access" confusion (the Planner role genuinely has no write_workspace_file tool by design — DeepSeek's own statement to that effect was accurate, not hallucinated). Full verdict and evidence trail in MODELS.md's new "Hosted" section entry.

2026-07-29: "no proper report despite solid findings" investigated end-to-end — one real coverage gap fixed, one real check-priority-shadowing bug fixed, both live-verified

User pushed back on the earlier 2026-07-29 completion-check cleanup session's own diagnosis that the findings/report-writing stage wasn't structurally overwhelming models, insisting on a fresh, whole-repo pass over every historical run rather than trusting the prior audit's conclusion. A dedicated catalog pass over ROADMAP.md's full History, RESEARCH.md, and every session_status/ *.md archive found 17 distinct report-writing-stage incidents across 5 patterns, spanning 2026-07-10 through 2026-07-28 and every model this project has ever tested, including the trusted gpt-oss:20b baseline. Two real, distinct root causes were fixed as a result — see Completed below (check_report_underuses_evidence, the check-priority-shadowing fix in completion.py/ grounding.py) — both grounded in checked literature (NAACL 2025 "Coverage-based Fairness in Multi-document Summarization", arXiv:2412.08795; "Regression Accumulation in Multi-Turn LLM Programming Conversations", arXiv:2607.01855) and both live-verified: the first via a real 48-minute gpt-oss:20b run against the standing 2-facet benchmark query, the second by feeding that SAME run's actual saved final_report.md into the fixed code directly and confirming the previously- silent uncited_claims problem now surfaces.

A live smoke test caught a real gap in my own verification, worth recording: the first "smoke test passed" read of that 48-minute run was wrong — it checked for crashes and false-positive regressions on the new check, but never read what the report actually said. The user rejected that read outright ("this is not acceptable, no proper report made") and was right to: the report used a fabricated (Source 1)...(Source 8) inline numbering scheme with no numbered reference list at the bottom (unverifiable to a human despite passing URL-presence grounding), a claim attributing payroll-tax coordination to a source that, read directly, never mentions it, and — the concrete bug fixed this session — check_uncited_claims never got a turn across 3 attempts because check_stub_source kept winning real_grounding_problem's own first-hit priority chain, and the terminal "retry budget exhausted" message reported only stub_source even though uncited_claims was independently, simultaneously true. Read the real output, not just the check log, before calling any smoke test conclusive.

2026-07-28: Ornith-1.0-9B bake-off — INCONCLUSIVE, two real DeepDelve architecture bugs found+fixed, one serving-layer gap isolated

Full trail in RESEARCH.md §14 (a-g); working summary in session_status/CURRENT.md. Candidate: deepreinforce-ai/Ornith-1.0-9B (dense, Qwen3.5-arch, MIT), the untested middle ground identified after gpt-oss:20b between the already-exhausted sub-14B local-model space and paid frontier APIs. GLM-4.7-Flash and Ornith-1.0-35B were both ruled out on hardware grounds first (smallest available quants 19GB/21.2GB, over this hardware's 17.1GB VRAM budget) without spending GPU time on either.

Five live runs (cold pull + benchmark, then three --resume-run attempts) surfaced a genuinely strong cold-start synthesis (45 real sources, correct architecture-family coverage matching the benchmark's own gold reference — the best findings.md of any candidate this project has tested), but never converged on a clean, fully-verified final_report.md. Root cause of the looping/ self-rejection pattern traced to the stock chat template's empty-<think>-injection defect, independently corroborated by deepreinforce-ai/Ornith-1's own GitHub issues (#4, #16) and two Reddit threads (r/LocalLLaMA) — a real, model-family-wide trait, not a DeepDelve-specific one. Fixed via froggeric/Qwen-Fixed-Chat-Templates, patched directly into the pulled GGUF's tokenizer.chat_template metadata (gguf_new_metadata.py --chat-template-file, no tensor rewrite).

Two real, model-independent DeepDelve architecture bugs found and fixed along the way (would affect any model hitting these paths, not specific to this candidate):

1. check_not_delegated was scoped to the current process's live quota pool (always 0 at the start of a resumed process), contradicting build_resume_input's own "don't re-delegate" instruction — live-confirmed to derail a resumed run into a think_tool reflection loop. Fixed: Ctx.delegated now also checks run_state.data["fetched_urls"] (already resume-carried-over ground truth). ARCHITECTURE.md §4 updated from "known, not fixed" to "fixed".
2. Builder was dispatched with grounding-check inject text worded for the Planner ("delegate a Searcher"), but Builder has no delegate_tasks tool at all — live-confirmed to get it stuck narrating "I will delegate..." instead of ever rewriting the file. Fixed: a shared _BUILDER_NO_DELEGATE_CLARIFICATION string appended to both Builder-dispatch branches. ARCHITECTURE.md §2 gained a new subsection on this class of bug.

Both fixes have regression tests in test_structural_checks.py; full suite passing.

Added edit_workspace_file (src/tools/fs.py, targeted old-string/new-string replacement, wired into both Builder and FindingsWriter — app.py, quota in config_template.yaml + live config, prompt text in prompts.py) after confirming live that a "drop 3 flagged citations, keep everything else" correction cycle was a genuine capacity edge for full-document regeneration (one attempt made zero write/edit calls across ~8 minutes; successive full rewrites fixed old stub citations while introducing new ones, never converging). Live-tested once after adding it — the model didn't spontaneously reach for the new tool even on a textbook case; inconclusive on one sample, not contradicted, left open for a future retest.

Real serving-layer gap isolated and confirmed (RESEARCH.md §14e, ARCHITECTURE.md §6, new): four direct API tests holding everything else constant showed thinking suppression works cleanly via Ollama's native /api/chat endpoint, even with tools present, but leaks via the OpenAI-compat /v1/chat/completions endpoint DeepDelve actually uses — specifically triggered by tools being present in the request. Since DeepDelve is built entirely on an OpenAI-compatible client, this likely affects every model ever tested through Ollama on this project whenever it makes a tool call with thinking nominally disabled, not just this candidate. Not fixed — proposed as a real architecture question (a backend-adapter abstraction: how does a serving backend want thinking toggled / tool schemas shaped / errors surfaced, replacing the growing pile of if base_url contains X special cases in _get_default_options()), not yet designed or scoped. Added to Pending below.

Process hygiene lesson: nohup python src/app.py ... & returns the bash wrapper's PID, not the actual Python process's — killing the wrapper alone left two real app.py processes orphaned in the background mid-session, each still holding a request against the model. Use pkill -f "resume-run <folder>" or capture the real child PID directly next time.

Verdict: Ornith-1.0-9B left INCONCLUSIVE, not DISQUALIFIED, not PASSED — every failure mode hit had an independent non-model explanation attached (two fixed DeepDelve bugs, one serving-layer gap, one missing tool now added), so no run tonight was a clean, unconfounded test of this candidate's real ceiling per the Model Evaluation Standard. A future clean re-test (proper process tracking, full attempt budget already in place) is the natural next step, not a repeat of tonight's confounded conditions. Ollama tags (deepdelve-ornith-9b, deepdelve-ornith-9b-froggeric) and the patched GGUF left in place, not cleaned up, since this isn't a closed disqualification.

2026-07-28: Fine-tuning resumed — comprehensive 7-dimension combined GRPO round trained, held-out-evaluated, confirmed real generalization

User explicitly decided to resume fine-tuning after the indefinite pause (see "Stretch" section's own standing note, updated alongside this entry), on the condition that every documented reward dimension be accounted for first rather than training with partial coverage again. Auditing finetune/reward.py against what finetune/train_combined_grpo.py actually trained (vs. what its own docstring merely documented) found 3 real gaps, all closed same-day: real_tool_name_reward (had real calibration data, tool_name.jsonl, but no synthetic-prompt generator, never trained); check_stale_findings and check_uneven_task_investment (no reward function at all). New generators built for all three (generate_synthetic_tool_name_prompts.py, generate_synthetic_stale_findings_prompts.py, generate_synthetic_uneven_investment_prompts.py), plus task_verification_flagged scenarios added to the existing generate_synthetic_prompts.py (today's own live VERIMAP quota-fix subject, reuses thin_coverage_response_reward — confirmed structurally identical response shapes, not assumed). Full design plan at ~/.claude/plans/imperative-tumbling-journal.md.

Training round: Qwen/Qwen3-4B base, LoRA (r=16), 7 task_types / 302 rows / 5 distinct reward functions (up from 3 task_types / 158 rows in the prior 2026-07-19 round) — the "one combined retrain, never piecemeal" methodology rule (below) honored fully this time, not partially like the 2026-07-21 attempt that added writer_role_response_reward but never actually launched.

Took 5 attempts to complete, 2 real bugs found and fixed via live training (both genuine model-output edge cases the reward functions weren't defensive against, not hypothetical):

1. findings_underuses_evidence_response_reward crashed on a None per-task URL list — root cause: datasets.Dataset.from_list unions every row's dict keys into one Arrow struct schema, padding keys absent from a given row with None (not an empty list) since different findings_evidence scenarios name different tasks. Fixed: treat None like [].
2. schema_compliance_reward crashed (AttributeError) when a real completion emitted "arguments" as a JSON array instead of an object — ironic, since this is exactly the malformed shape that function exists to score 0.0 for. Hit at step 250/260 (96% through a ~95-minute run, save_strategy="no" meaning nothing was saved yet — a full restart). Fixed once at the root with a new shared _tool_args() helper (not patched separately at the 4 call sites that had the same unguarded tool_call.get("arguments") or {} idiom). Checkpointing (save_steps=50, save_total_limit=2, --resume flag) added afterward so a future crash costs at most ~50 steps, not the whole run.

Held-out evaluation (finetune/evaluate_combined.py, extended today from 2 dimensions to all 7 — was a real pre-existing gap, findings_evidence had no held-out coverage either before today): 39 held-out prompts, topics never in the 302 training rows.

Dimension	base	fine-tuned	delta
thin_coverage	0.542	1.000	+0.458
citation_grounding	0.615	0.781	+0.167
findings_evidence	0.562	1.000	+0.438
tool_name	0.925	1.000	+0.075
stale_findings	1.000	0.875	-0.125
uneven_task_investment	1.000	1.000	0.000
task_verification_flagged	1.000	1.000	0.000
combined overall	0.747	0.926	+0.179

Verdict: real generalized improvement confirmed, not memorization — the two hardest, most load-bearing dimensions (thin_coverage, findings_evidence) improved substantially on topics the model never trained on, directly answering the overfitting question raised mid-session. Real caveats: stale_findings regressed (-0.125) but on only 2 held-out prompts, plausibly noise; citation_grounding (the dimension most tied to actual hallucination risk) improved but did not reach ceiling (0.781) — the residual gap here matters most of any dimension.

NOT YET DONE — this is still an offline reward-function score, not a live-verified result. Adapter saved to /mnt/nuevovol/llm-models/qwen3-4b-combined-v2-lora (a NEW directory, the prior 2026-07-19 2-dimension LoRA at qwen3-4b-combined-lora was deliberately left untouched pending this round's own validation). Before this counts as deployable: (1) load the LoRA into a real DeepDelve run and benchmark it live, same standard every model candidate is held to, (2) specifically re-test a citation_grounding-shaped scenario given the residual gap, (3) decide whether the stale_findings regression is real (more held-out prompts for that one dimension) or noise. Full session detail in session_status/2026-07-28.md once archived (see CURRENT.md if not yet archived).

2026-07-28 (later): v2 combined LoRA live-benchmarked end-to-end — DISQUALIFIED

Closes the "NOT YET DONE" caveat on the entry directly above. Deployment pipeline: merge_and_unload() (CPU) → GGUF q8_0 via convert_hf_to_gguf.py (4.27GB) → Ollama tag deepdelve-qwen3-4b-combined-v2 (num_ctx 40960, reused the proven qwen3:4b tool-call template) → direct tool-call smoke test passed cleanly → ~/.deepdelve/config.yaml pointed at it → live run against the standing sales-forecasting/heuristic-algorithms benchmark (eval/sales_forecasting_benchmark.md), same query every prior model candidate has been run against.

Result: DISQUALIFIED — final_report.md never written, retry budget exhausted. Run folder: research_output/i_want_documentation_on_heuristic_algoritms_for_de_20260728_143953/. check_task_verification_flagged fired twice against the same 3 tasks for fabricated/unverified citations; the model's own first synthesis attempt narrated report content in prose instead of calling write_workspace_file ("Since I cannot write or edit files directly, I will describe the content..."). After 2 failed grounding-verification retries it re-dispatched a duplicate/untracked task, tripping check_untracked_delegation's hard stop before any report was produced. This directly answers the open "live citation_grounding retest" question from the entry above: still broken live — consistent with the held-out eval never reaching ceiling (0.615→0.781). The stale_findings open question was not exercised (no report to compare).

Model Evaluation Standard point 3 note: confirmed via raw API test, on both this candidate and plain qwen3:4b, that the run executed under the already-known, already-accepted Qwen3 think-mode-passthrough Ollama bug (chat_template_kwargs.enable_thinking:false + reasoning_effort:none suppressed nothing on either tag) — a pre-existing, model-family-wide confound (see the "Ollama restored" entry below), not something specific to this LoRA. Named for the record, doesn't change the verdict: base qwen3:4b carries the identical confound and this candidate still fabricated citations and never produced a report.

Cleanup same batch (per this project's disqualified-candidate hygiene rule): ollama rm deepdelve-qwen3-4b-combined-v2, deleted qwen3-4b-combined-v2-merged (7.6GB) and qwen3-4b-combined-v2.q8_0.gguf (4.0GB), ~11.6GB reclaimed. Kept the LoRA adapter itself (qwen3-4b-combined-v2-lora, 132MB) as the training artifact. config.yaml restored to deepdelve-gpt-oss:latest.

2026-08-19: v2 combined LoRA re-tested clean through api.backend: "ollama" — DISQUALIFIED, not confounded this time

Closes the "clean re-test, not yet done" caveat this entry carried since 2026-07-28. Re-merged the still-on-disk LoRA adapter (merge_and_unload() on CPU → convert_hf_to_gguf.py --outtype q8_0, 4.27GB) → redeployed as deepdelve-qwen3-4b-combined-v2 via llama.cpp-convert (already set up on /mnt/nuevovol/llm-models) → confirmed via direct curl against the live tag that think: true (NOT false) correctly isolates reasoning into its own field on the native /api/chat endpoint, think: false reproduces the identical known bug. Ran one clean trial via eval/evaluate.py --config pointed at a dedicated config (backend: ollama, enable_thinking: true, model deepdelve-qwen3-4b-combined-v2) against the standing visa/rent ablation benchmark.

One real process mistake along the way, worth recording: the first attempt at this re-test edited the LIVE ~/.deepdelve/config.yaml and launched eval/evaluate.py WITHOUT --config — evaluate.py never reads the live config at all, it always builds its own from src/tools/config_template.yaml unless --config is passed explicitly (_resolve_base_config_path, eval/evaluate.py:77). That entire ~47min run silently re-benchmarked deepdelve-gpt-oss again, not the candidate — a wasted run, caught only by checking the run's own eval_agent_config.yaml after the fact. Corrected by building a dedicated eval/qwen3-4b-combined-v2-config.yaml and passing it via --config, confirmed correct by re-checking the new run's own materialized config before trusting the result.

Result: score 0.25 (worse than the confounded run's 0.5), run finished on its own in 1513.8s (well under the 2820s ceiling, no resume needed). final_report.md is the deterministic-salvage banner — the model narrated in chat instead of ever calling write_workspace_file across its full writer-retry budget. findings.md repeatedly failed grounding on the same fabricated URLs (rentremote.com, nomadsembassy.com) across consecutive rebuild attempts, the same citation- fabrication shape as the original 2026-07-28 disqualification. Verdict confirmed, cleanly: the fine-tune's targeted objectives are real (held-out gains hold), but citation fabrication and writer-dispatch convergence — dimensions the combined reward never targeted — are still broken at 4B scale even with reasoning cleanly isolated from .content. Not a serving-layer artifact this time; a real capability ceiling for this base size on these specific failure modes.

Cleanup: ollama rm deepdelve-qwen3-4b-combined-v2, deleted the re-generated qwen3-4b-combined-v2-merged and qwen3-4b-combined-v2.q8_0.gguf, ~/.deepdelve/config.yaml restored to deepdelve-gpt-oss:latest/backend: openai/enable_thinking: false. LoRA adapter kept (training artifact). eval/qwen3-4b-combined-v2-config.yaml kept in the repo as a reusable template for the next candidate's clean re-test, since building it correctly (targeting the right line, not a comment that happens to contain the same substring) took one real mistake to get right too.

Root-caused same day: retries "not recovering" was NOT a model-capability problem — it was a structural bug. The known Qwen3 think-passthrough Ollama bug (see below) inflated every turn's token/char count 2-3x, blowing through context_budget_chars (50000, calibrated for nothink operation) in just 2 of the 8 configured completion-check attempts. tui.py's context-budget force-final path (run_state.attempt = 10**6) then jumped straight to run_completion_check's terminal branch, where _yield_to_starved_check's starvation window happened to be active and swapped the real, still-retriable problem (task_verification_flagged, citation fabrication) out for check_untracked_delegation — a check whose own docstring explicitly promises it "will NOT block this run from finishing." The forced-final path had no awareness of that contract, so a low-severity hygiene nudge got reported as the run's terminal blocker instead of the real correctness problem, which never actually got its intended 8-attempt budget to resolve.

Fixed (src/engine/completion.py, _yield_to_starved_check): added a never_final_blocker parameter, True for the check_untracked_delegation call site only (check_report_underuses_findings, a genuine correctness signal with no such guarantee, is unaffected — defaults False). Once ctx.attempt >= ctx.max_attempts (i.e. this cycle is already headed to the final branch regardless), the starved-check override is skipped so a documented-non-blocking check can never become the reported terminal reason. New regression case added to test_structural_checks.py's existing _starvation_guard_scenario; full suite re-run and passing.

Consequence for the DISQUALIFIED verdict above: it stands as recorded, but is now understood to be confounded — the model never got a real chance to resolve the citation-fabrication problem before being cut off. Re-benchmarking with this fix in place (and, ideally, once the Qwen3 think-passthrough bug itself is resolved or worked around) would be needed for a clean verdict; not done this session per explicit scope ("fix the bug, don't re-benchmark yet").

2026-07-26 (final entry, end of day): Ollama restored as the permanent serving backend, vLLM removed

Reverses the 2026-07-21 "Ollama dropped" decision, after the same-day vLLM re-test sweep above disqualified/discarded 9 candidates in a row, several with genuinely serving-layer-shaped symptoms (a silent zombie crash, intermittent empty-JSON tool calls, a garbled tool-call marker). Research (RESEARCH.md §11, primary-sourced against vLLM's own PR history and Feb 2026 blog, not just inferred from symptom shape) found: vLLM+bitsandbytes-on-ROCm support is real but young specifically on consumer RDNA GPUs (this hardware ran bitsandbytes exactly at its minimum required ROCm version floor); llama.cpp's HIP backend (what Ollama uses) turned up zero documented quantization-correctness issues across the same research, with years more real-world maturity on consumer AMD hardware.

Important, deliberately not oversimplified: most of today's actual DISQUALIFIED verdicts were NOT ROCm-caused (thin_coverage/narrate-instead-of-call reproduced identically on Ollama-hosted candidates too, including qwen3:8b on both backends) — this reverts the SERVING BACKEND to fix the flaky/crashy low-level tail, not a claim that it changes any model-capability verdict already reached. The two bugs that originally motivated dropping Ollama (Qwen3 think-mode passthrough, ollama/ollama#6155) are still real and unfixed — this is an explicit, informed tradeoff (accept those two known, narrowly-scoped bugs) in exchange for llama.cpp/HIP's overall serving-layer maturity edge on this specific hardware, not a claim that Ollama is now bug-free.

Cleanup: ~/.venvs/vllm deleted (~11GB), vLLM-specific HF cache checkpoints deleted (openai/gpt-oss-20b, ~13GB — DeepDelve's own actual dependencies, bge-reranker-v2-m3/ nli-deberta-v3-small/all-MiniLM-L6-v2, sharing the same HF_HOME cache location, deliberately preserved, NOT deleted). ~/.deepdelve/config.yaml restored to http://localhost:11434/v1/deepdelve-gpt-oss:latest, settings.skip_chat_template_kwargs reset to false. Full detail in the project_ollama_restored memory (supersedes project_ollama_dropped).

2026-07-26 (later still again): mistral-nemo:12b re-tested with the Mistral fix — DISQUALIFIED, same thin_coverage non-convergence pattern

Closes out the original 2026-07-21 BLOCKED verdict (see its own entry above, which already noted this exact fix as the remaining open re-test opportunity). Checked real weight size first, per the lesson from qwen2.5-coder/devstral: HF repo's mistralai/Mistral-Nemo-Instruct-2407 lists two packagings of the same ~24.5GB bf16 weights (consolidated single-file + 5-shard set); at the observed 4-bit ratio this comfortably fits (~8.3GiB actual, confirmed — matches the original pre-flight spike, which used this exact model). bitsandbytes 4-bit, mistral tool parser, settings.skip_chat_template_kwargs: true (today's fix, commit 9c8111f) — request reached the model cleanly this time, no 400.

Isolated tool-call smoke test: 6/8 clean across two batches (75%) — real structured arrays, no #6155-class bug — with two distinct intermittent failure shapes (a garbled tool-call marker prefix once, one full narration — "Cerebro has delegated the following tasks..." — with no real call once). Better isolated reliability than most other candidates today, not perfect.

Full DeepDelve benchmark run (the standing stress-test query): real, genuine engagement with the task — 13 sources fetched, 0/7 web-search failures, 8 findings recorded — clearly not a narrate-only or zero-engagement failure. But ended Retry budget exhausted (thin_coverage), final_report.md never written, all 4/4 completion-check attempts hitting the IDENTICAL thin_coverage problem — the same non-convergence signature already documented 3 times this project across different models AND backends (qwen3:4b on Ollama, qwen3:8b on both Ollama and vLLM): once thin_coverage fires, the Planner doesn't act on the corrective nudge and keeps repeating similar coverage instead of genuinely redelegating deeper. This is well-corroborated evidence even from one live run of THIS specific candidate, given how consistently this exact pattern recurs across unrelated model families and serving backends.

Verdict: mistral-nemo:12b DISQUALIFIED. The original 2026-07-21 BLOCKED status is now resolved — the chat_template_kwargs infrastructure issue is confirmed fixed (real research happened this time), but the underlying capability question comes back negative, joining the thin_coverage non-convergence family rather than escaping it. Not the same failure class as mistral:7b-instruct's own same-family verdict (not_delegated, narrate-instead-of-call) — this candidate genuinely engages with the task, it just can't escape thin-coverage retries once triggered.

Cleanup: vLLM server SIGTERM'd cleanly (zero orphan VRAM/processes confirmed), HF cache checkpoint deleted (~23GiB — this is a CLOSED, final verdict, unlike qwen2.5-coder's deliberately-preserved open item), ~/.deepdelve/config.yaml restored to the gpt-oss baseline.

2026-07-26 (later once more): devstral:24b discarded on hardware grounds without a pull attempt

Checked real weight size FIRST this time, per the lesson just learned from qwen2.5-coder:14b-instruct (don't spend time before confirming it fits). mistralai/Devstral-Small-2507's HF repo lists 94.3GB of .safetensors files, but that's misleading: it contains BOTH a single-file consolidated.safetensors (47.14GB, for mistral-inference) AND a separate 10-shard set totaling the same ~47.1GB (for HF transformers/vLLM) — the same weights packaged twice, not two different models. vLLM would only load the sharded set, so the real relevant footprint is ~47.1GB bf16.

Applying the empirically-observed bitsandbytes 4-bit compression ratio from qwen2.5-coder-14b earlier today (27.5GB bf16 → 9.9GB actual VRAM footprint, ≈0.36×): devstral's 47.1GB bf16 would land around ~17GB quantized — exceeding the entire 17.1GB card before any KV cache or activation overhead is even counted. Discarded on hardware grounds, same standard as qwen3.6's earlier discard — no pull attempted, no GPU time spent.

2026-07-26 (still later): qwen2.5-coder:14b-instruct — INCONCLUSIVE, wrong parser used initially, correct parser found but shows ~50% unreliable extraction

Qwen/Qwen2.5-Coder-14B-Instruct (confirmed real, not gated, via direct HF API check), bitsandbytes 4-bit. This candidate went through three distinct rounds before landing on an honest verdict — each correction below was caught by the user pushing back on an over-confident claim, not found independently, worth recording as-is.

Round 1 — hermes tool parser, crash, initially misdiagnosed. First launch at gpu_memory_utilization 0.7 failed cleanly with a graceful ValueError (needs 3.0 GiB KV cache, only 0.44 GiB available). Bumped to 0.9 (~14.3GiB budget) and retried — weight loading succeeded ("Model loading took 9.9 GiB"), then the process crashed silently with zero further log output (no "Available KV cache memory" line, which every other candidate today logged within ~2 seconds of the same point) — EngineCore found in zombie (Z) state, parent APIServer hung forever, no traceback anywhere. Initially attributed to a system-RAM OOM (the log's own "checkpoint size exceeds 90% of available RAM" warning looked suggestive) — user asked "are you sure that's the reason?", which prompted actually reading weight_utils.py's source: that warning is about an OPTIONAL read-ahead prefetch for network filesystems, correctly skipped here (local NTFS mount), its absence being the RAM-safe path, not a risky one. That theory didn't hold. Discarded the candidate on hardware-margin grounds instead (weights alone = 9.9GiB of ~15.9GiB usable budget).

Round 2 — user asked for a retry; the crash didn't reproduce. A clean second launch (same flags) got PAST the exact point that crashed before — "Available KV cache memory: 3.62 GiB", "Application startup complete". The crash was transient/intermittent, not deterministic — the "discard on hardware grounds" verdict from Round 1 was wrong on its own terms (it DOES start and run, just tightly). Ran the isolated delegate_tasks-shaped smoke test against the now-live server: 3/3 consistent failures, but a NEW, different shape than any other candidate today — the model produced a genuinely well-formed, correctly-structured tool call, just wrapped in <tools> tags instead of the <tool_call> tag the hermes parser requires.

Round 3 — wrong parser confirmed via primary sources, correct one found and installed. Web research + direct inspection of Qwen/Qwen2.5-Coder-14B-Instruct's own chat template confirmed: Qwen2.5-Coder (unlike vanilla Qwen2.5-Instruct) was never trained on the Hermes <tool_call> convention — it uses <tools> tags, and vLLM has no native parser for this variant (confirmed via direct grep of the installed tool_parsers/__init__.py registry). A real community project exists specifically for this gap: hanXen/vllm-qwen2.5-coder-tool-parser (12 stars, actively maintained, pushed 2026-04-29). Reviewed the actual parser source before installing it (392 lines, mirrors vLLM's own bundled parser structure, no network/subprocess/eval calls) — safe. Relaunched with --tool-parser-plugin/--tool-call-parser qwen2_5_coder/its accompanying --chat-template. Smoke test with the CORRECT parser: 2/4 consistent successes (real structured array), 2/4 returned arguments: "{}" (empty) despite finish_reason: "tool_calls" and a similar completion-token count to the successful runs — the plugin's own README independently documents that longer system prompts degrade this model's JSON-generation quality even under ideal conditions (49/50 vs 50/50 in their own "explicit" vs "minimal" mode tests), which is at least directionally consistent with an unreliable-under-load pattern, though the empty-{} extraction itself could be either the model or the third-party parser's own bug — not disambiguated, and not chased further given the time already spent on this one candidate.

Verdict: INCONCLUSIVE, not DISQUALIFIED and not a clean pass. Real capability under the correct format was never cleanly established either way — ~50% extraction reliability in isolated testing is too unreliable to call a pass, but it's also not the kind of confident, repeatable failure (not_delegated, fabricated content, #6155-class bug) other candidates today showed. A full DeepDelve benchmark run was never attempted (would need the custom chat-template/parser wired into ~/.deepdelve/config.yaml's vLLM endpoint, which only needs the server-side flags already proven above — no DeepDelve-side changes). Left as a genuine open item, not closed.

Cleanup: vLLM server SIGTERM'd cleanly (zero orphan VRAM/processes confirmed each round), HF cache checkpoint deleted (~28GiB) after the final round.

2026-07-26 (yet later): hermes3:8b DISQUALIFIED — narrates fake system messages instead of calling tools

Continuing the same day's vLLM re-test track. NousResearch/Hermes-3-Llama-3.1-8B, bitsandbytes 4-bit, hermes tool parser (confirmed correct, verified not gated via direct HF API check first). Isolated tool-call smoke test with the real delegate_tasks-shaped nested-array schema PASSED cleanly 3/3 (real structured tasks array, no #6155-class bug). No thinking mode in this model's chat template (checked directly, both default and tool_use template variants) — nothink is N/A for this candidate, same as llama3.2:3b/qwen2.5:3b-instruct.

The real DeepDelve benchmark run reveals a genuinely bizarre, distinctive failure: on its very first delegate_tasks attempt, the model narrated a FABRICATED system error as plain response text — "It looks like the prompt exceeded the maximum context length allowed by the model... I will try again with a shorter prompt." — then repeated this fake-retry narrative twice more, never once calling a real tool. Confirmed this error is entirely invented, not a real vLLM rejection: grepped the vLLM server's own log for the same time window — no context-length error of any kind exists there, and GPU KV cache usage was only 1-13% at that point, nowhere near the 16384-token ceiling. The model fabricated a plausible-sounding excuse out of nothing.

Confirmed via a second, independent run (a deliberately simple, different query, "what is the speed of light" — to rule out query complexity as a confound): same not_delegated root cause, different specific narration — this time the model wrote literal .delegate_tasks(tasks=[...]) pseudo-code as prose text, and after DeepDelve's own completion check explicitly told it "No delegate_tasks call was ever made," the model apologized in text ("Apologies for the repeated attempts. I understand the system's warning now.") and then immediately narrated the exact same fake call as text AGAIN instead of actually invoking the tool. Meets Model Evaluation Standard point 4 (more than one run): two independent runs, different queries, same not_delegated root cause each time, with two different specific hallucination shapes.

Verdict: hermes3:8b DISQUALIFIED. Same broad failure CLASS as mistral:7b-instruct's own same-day verdict (narrate instead of call, only surfacing under DeepDelve's real, much longer system prompt — the isolated short smoke-test prompt cannot catch this), but a more severe, specific manifestation: fabricating entirely fictional system/error text rather than just paraphrasing the intended call. Not a serving-layer issue — the isolated smoke test against the same endpoint/parser was clean.

Cleanup: vLLM server SIGTERM'd cleanly (zero orphans confirmed), HF cache checkpoint deleted (~15GB), ~/.deepdelve/config.yaml restored to the gpt-oss baseline.

2026-07-26 (even later): settings.skip_chat_template_kwargs fix unblocks Mistral-family vLLM candidates; mistral:7b-instruct DISQUALIFIED (narrate-instead-of-call, not the chat_template block)

Continuing the same day's vLLM re-test track. mistral:7b-instruct (mistralai/Mistral-7B-Instruct-v0.3, bitsandbytes 4-bit, mistral tool parser — same proven path as the original pre-flight spike) passed its isolated tool-call smoke test cleanly (3/3, real structured tasks array, no #6155-class bug — a genuine improvement over the original Ollama-era "rejected at schema stage" verdict). But the real DeepDelve benchmark run failed immediately with the SAME 400: "chat_template is not supported for Mistral tokenizers" error that already marked mistral-nemo:12b BLOCKED — and that entry had already predicted this would recur for mistral:7b-instruct and devstral:24b, now confirmed live for the first.

Root-caused directly at the vLLM source, not just re-confirmed from the earlier BLOCKED entry: vllm/tokenizers/mistral.py:161-162 — if request.chat_template is not None or request.chat_template_kwargs is not None: raise ValueError(...). Confirmed via vLLM's own PR history this is INTENTIONAL and permanent, not a version-fixable bug: PR #26358 (merged Oct 2025, well before this install's 0.25.1) explicitly changed this from a warning to a hard error, because Mistral's own mistral-common tokenizer library doesn't support overriding its built-in chat template in native Mistral tokenizer mode at all — checked both the model's own HF card (no vLLM serving guidance at all) and vLLM's docs/GitHub issues before concluding this; no server-side flag or --tokenizer-mode variant avoids it (mistral-nemo's own investigation already ruled out auto and hf modes).

Fixed on DeepDelve's side (commit 9c8111f): new settings.skip_chat_template_kwargs (default false), checked in _get_default_options() (src/engine/orchestrator.py) — when true, skips building extra_body (and therefore chat_template_kwargs/reasoning_effort) entirely. Explicit, user-set config, not model-family auto-detection (this project has repeatedly found string-matching heuristics unreliable — same philosophy as settings.specialist_model's own explicit design). Unblocks all three Mistral-family vLLM candidates at once (mistral-nemo:12b, mistral:7b-instruct, devstral:24b), not just this one.

With the fix applied, mistral:7b-instruct's real benchmark run got PAST the 400 wall — and hit a different, genuine capability failure: not_delegated, two independent runs, different queries, identical result. Run 1 (the standard stress-test query, "top 5 heuristic algorithms" + "Colombian holiday spending"): 4/4 completion-check attempts all not_delegated, zero findings, zero sources fetched, 145s. Run 2 (deliberately simple, "what is the speed of light," to rule out query complexity as a confound): identical not_delegated failure, zero sources, 77s. Root cause, confirmed from the real session log content: the model consistently NARRATES its planned delegate_tasks call as a literal markdown fenced code block in its response text (e.g. ```\ndelegate_tasks(tasks=[...])\n```) instead of actually emitting a structured tool call — even under --enable-auto-tool-choice. This reproduces the same "narrate instead of call/write" failure class already documented for Bonsai-8B and qwen2.5:3b-instruct (those at the FindingsWriter/writer-role stage; this one at the Planner's very first delegate_tasks, immediately) — DeepDelve's real, much longer system prompt evidently pushes this model toward narration in a way the short isolated smoke-test prompt didn't trigger. Meets Model Evaluation Standard point 4 (a discard claim needs more than one run): two independent runs, different queries, identical failure.

Verdict: mistral:7b-instruct DISQUALIFIED — genuine capability gap, confirmed NOT the chat_template_kwargs infrastructure block (that's fixed and confirmed working: the 400 is verifiably gone, the request reaches the model successfully both times). Distinct from mistral-nemo:12b's own verdict, which remains BLOCKED-not-yet-re-tested (the fix removes its blocker too, but it hasn't actually been re-run with the fix applied — that's a separate, still-open opportunity, not assumed to share this exact verdict).

Cleanup: vLLM server SIGTERM'd cleanly (rocm-smi --showpids confirmed zero orphans), HF cache checkpoint deleted (~14GB, unquantized bf16 cache — larger than the 4-bit runtime footprint since bitsandbytes quantizes on load, not on disk). ~/.deepdelve/config.yaml restored to the gpt-oss baseline (skip_chat_template_kwargs: false, openai_model: deepdelve-gpt-oss:latest).

2026-07-26 (later still): vLLM bake-off re-test resumed — qwen2.5:3b-instruct and llama3.2:3b both DISQUALIFIED, one result challenges the track's own core premise

Continuation of the same day's session, resuming the vLLM re-test plan (~/.claude/plans/moonlit-plotting-simon.md) after mistral-nemo:12b (BLOCKED), llama3-groq-tool-use:8b (DISQUALIFIED), and qwen3:8b (DISQUALIFIED) were already closed in earlier sessions. qwen3.6 (35b-a3b) was checked and discarded outright on hardware grounds before spending any GPU time — confirmed via a direct HF API blob-size query that its FP8 checkpoint alone is 37.5GB, more than double this hardware's entire 17.1GB VRAM budget, and no 4-bit pre-quant exists; even a hypothetical 4-bit re-quant (~18.7GB) still wouldn't leave real KV-cache headroom. Per Model Evaluation Standard point 6, discarded without attempting a pull.

• Qwen/Qwen2.5-3B-Instruct — DISQUALIFIED, new failure mode found on vLLM, not the original Ollama-era verdict's own reason. No quantization needed (bf16, ~6GB, easy fit). Tool parser: hermes (confirmed correct — qwen3_engine_tool_parser variants are Qwen3-only, don't apply to 2.5). Isolated single-arg tool call (web_search) parsed cleanly. A nested-array schema matching delegate_tasks' real shape (a tasks array of {task_name, instructions, agent_id} objects) consistently failed, 5+ independent reproductions: the model reliably emits genuinely malformed JSON for its own tool call — closes the "arguments" object but omits the FINAL closing brace for the outer {"name": ..., "arguments": {...}} wrapper (finish_reason: "stop", not "length" — not a token-budget cutoff, the model just stops one brace short). Verified this is a genuine model-generation defect, not a vLLM parser bug: tested hermes_tool_parser.py's extract_tool_calls directly in isolation against the real captured response text — the parser's own regex extraction is correct (confirmed via regex.findall on the raw content), it's the extracted JSON substring itself that's malformed at the source. Consistently reproduced (open brace count 4, close brace count 3) across independent live requests, not a one-off sampling fluke. This is a DIFFERENT failure mode than the original Ollama-based verdict (FindingsWriter never successfully calling write_workspace_file across 8 attempts) — same overall verdict (disqualified), new independent evidence via a completely different backend.

• meta-llama/Llama-3.2-3B-Instruct — DISQUALIFIED, and this result is significant beyond just this one candidate: it directly challenges the vLLM re-test track's own founding premise. Gated repo — required a real HF access-request-and-approval cycle mid-session (documented for process continuity: an hf auth login-verified valid token can still 403 on the actual /resolve/main/... download endpoint despite the account showing normal "Use this model" UI on the model page; the metadata API's visible file listing is not proof of real download rights — only the resolve endpoint is ground truth). Tool parser: llama3_json (confirmed via vllm/tool_parsers/__init__.py). Isolated tool-call smoke test with the exact same delegate_tasks-shaped nested-array schema used for the Qwen2.5 test above: tasks came back as a JSON-encoded STRING inside the arguments object, not a real array — the textbook #6155 shape (ollama/ollama#6155, "Support Nested Parameters for Tools") — reproduced 4/4 times, fully consistent. But this ran on vLLM, not Ollama. The entire premise motivating this multi-session vLLM re-test effort was that #6155 is an Ollama-serving-layer bug that may have wrongly disqualified mistral-nemo:12b/llama3-groq-tool-use:8b/llama3.2:3b on a backend artifact rather than genuine model incapability — this result directly disproves that hypothesis for llama3.2:3b specifically: the identical stringified-array shape reproduces independently on a completely different serving stack (vLLM's own llama3_json parser, not Ollama's tool-call handling at all). Confirmed via vLLM's own official documentation (docs.vllm.ai/en/latest/features/tool_calling.html, "Known issues" section for Llama Models): "The model can generate parameters in an incorrect format, such as generating an array serialized as string instead of an array." vLLM's own maintainers attribute this to the MODEL's generation behavior, not something their parser introduces or could fix — this is Llama 3.2's own limitation, independent of serving backend. mistral-nemo:12b and llama3-groq-tool-use:8b already have their own independent, non-#6155 disqualification reasons (real infra incompatibility and missing XML wrapper tags respectively), so this doesn't overturn either of those verdicts, but it does mean the #6155-is-Ollama's-fault framing that motivated re-testing all three should be treated as disproven for the one candidate where a clean head-to-head comparison was actually possible, not assumed to still be an open question.

Cleanup: both candidates' vLLM servers SIGTERM'd cleanly (rocm-smi --showpids confirmed zero orphan EngineCore processes both times), both HF cache checkpoints deleted immediately after each verdict (Qwen2.5-3B-Instruct ~5.8GB, Llama-3.2-3B-Instruct ~6.0GB) rather than batched to session end — per explicit user instruction this session to reclaim disk space right after disqualifying a candidate, not just at the end.

vLLM re-test track status after this session: mistral-nemo:12b BLOCKED, llama3-groq-tool-use:8b DISQUALIFIED, qwen3:8b DISQUALIFIED, qwen3.6 discarded on hardware grounds without testing, qwen2.5:3b-instruct DISQUALIFIED, llama3.2:3b DISQUALIFIED. Remaining untested: gpt-oss:20b (deliberately deprioritized to last per explicit user instruction — it's the already-trusted default, lowest information value to re-confirm), the schema-stage rejects (devstral, hermes3, qwen2.5-coder, mistral:7b-instruct), the writer-role failures (the other Gemma-4-12B, granite3.1-dense, phi4-mini), and Bonsai-8B/the GRPO fine-tune (lowest priority per the plan's own "attempt only if time remains" framing).

2026-07-26 (later): three more production bugs + yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF DISQUALIFIED

Continuation of the same day's session below. All committed and pushed to main.

Three more real bugs found via live testing, each caught by actually reading final_report.md after a smoke test, not just confirming a mechanism fired (see this session's own hardened standing rule: a completion check firing correctly is not the same as the report being correct):

• 5fab261 — real_grounding_problem/related checks in src/utils/grounding.py compared cited URLs against fetched URLs with plain string equality. A citation the model percent-encoded (%E2%80%93) didn't match a fetched URL stored with the raw Unicode en-dash, false-flagging a genuinely correct citation as hallucinated. Root cause of a real run ("explain the main theories for the extinction of the dinosaurs") coming back with an EMPTY report — "No extractable findings were identified" — despite 27 real sources fetched. Added _normalize_url (percent-decode + rstrip), applied at every URL-comparison site in the module. Regression test pinned against the exact real citation/URL pair from the incident; confirmed failing on pre-fix code via git stash, passing post-fix.
• 6edd9ee — delegate_tasks is one shared closure/quota across every tier (Planner's top-level dispatch and every Tier-2 specialist's own dispatch to its Analyzer children draw from the same global tool_quotas_ctx counter, no per-task sub-budget anywhere). A live smoke test on a trivial single-fact query ("what is the Cretaceous–Paleogene boundary") saw ONE WebSearcher task call delegate_tasks 6+ times in a row, despite its own prompt saying "ONE authoritative source is sufficient" and "STOP EARLY" — consuming most of the run's entire global budget on one facet. Added settings.specialist_delegation_cap (default 3), enforced via delegation_depth_ctx (> 0 == a Tier-2 specialist, never the Planner) in src/engine/orchestrator.py. A real bug was caught in the fix itself before committing, via the same live-testing discipline: the first version used ctx.set(new_int) to update the counter across separate delegate_tasks calls, which silently never accumulated (8 Analyzer children spawned from one task, zero rejections) — the SDK dispatches each tool call as a separate asyncio.Task, so a .set() inside one call only mutates that Task's own context copy. Fixed by mirroring task_fetched_urls_ctx's existing mutable-list pattern (set once per dispatch, mutated in place). A second live run confirmed the cap actually holds.
• d88e445 — check_task_verification_flagged (VERIMAP Phase 1, shipped earlier the same day) had no notion that a flagged task could be superseded when the Planner renames it on retry instead of reusing the task_name. A live run got a flagged task renamed 3x ("...definition" → "...definition (narrow)" → "...definition (peer-reviewed)" → "...definition (peer-reviewed source)"); two renamed variants actually succeeded (verified, real Wikipedia-backed findings), but the check kept re-nudging the ORIGINAL stale names specifically — burned the entire completion-check retry budget (8/8), zero report ever written, worse than the bug this session started from. Fixed: _update_task_verification now does a second pass, downgrading a flagged entry to a new "superseded" status when its dispatched instructions (from run_state.data["dispatched_tasks"]) closely match an already-verified task's instructions, reusing the existing _looks_like_renamed_task helper. Also reworded the check's own directive to say "reuse the EXACT task_name" (defense in depth, matches the wording already applied to two other checks earlier the same day — soft instructions alone don't reliably hold on this hardware, which is why the structural supersede detection is the primary fix). Live-verification status: PARTIAL. A third live run of the same query showed NO repeat of the stale-nudge loop (real improvement), but the run was still going after 35+ minutes (Planner-level over-fanning into multiple top-level research angles — a separate, unrelated, still-open problem neither this fix nor the delegation cap constrains) and was killed before reaching a final report. Mechanism-confirmed (no repeat loop), not full-outcome-confirmed (never saw a final_report.md from this exact fix). Re-verify with a full run if this failure mode recurs.
• e60e334 — found while doing Model Evaluation Standard point 1 (confirm operating mode via a raw API test) on the Gemma candidate below, but it's a general engine fix: _get_default_options (src/engine/orchestrator.py) now also sends reasoning_effort: "none" alongside the existing chat_template_kwargs.enable_thinking whenever thinking is disabled — some models (see verdict below) ignore the latter entirely over Ollama's OpenAI-compat endpoint but respect the former. Confirmed harmless against the gpt-oss baseline (unchanged short harmony-format reasoning note).

New, separate, not-yet-acted-on finding: _stub_reason did NOT flag a ScienceDirect fetch that came back as almost entirely Adobe-Analytics tracking-script JSON, not real article content — no stub key at all in fetched_urls. Unlike a google.com fabricated-citation fetch in the SAME run, which WAS correctly caught as a stub. A real gap in _stub_reason's pattern coverage for this specific bot-wall shape; low priority, didn't actually poison a citation this run.

yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF — DISQUALIFIED, run twice, identical failure signature both times. Not the same candidate as the existing "Gemma 4 12B" row below (that one is SetneufPT/Gemma4-12B-IT-QAT_Q4_64K_16GB-GPU, a different publisher/quant of the same base architecture) — this is a separate fine-tune, evaluated separately.

• Base: google/gemma-4-12B-it, 11.9B params. Pulled as hf.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF:Q4_K_M (7.4GB). Derived tag deepdelve-gemma4-12b:latest created with PARAMETER num_ctx 16384 (the raw pulled tag has no context override — Ollama would otherwise silently serve it at its small default, invalidating any test per this project's own context-floor rule). OLLAMA_NUM_PARALLEL=1 already set system-wide, applies automatically.
• Model Evaluation Standard point 1, confirmed via direct curl against the real /v1/chat/completions endpoint before any benchmark: plain completion clean; tool-calling clean (correctly-shaped tool_calls, empty content). chat_template_kwargs.enable_thinking: false does NOT suppress thinking — the reasoning field stayed populated regardless (same passthrough-bug class already documented for Qwen3). Root-caused to a working alternative: reasoning_effort: "none" on the SAME endpoint DOES cleanly suppress it — this is what motivated e60e334 above. Confirmed clean (no reasoning field, correct tool_calls) with the fix applied, before running any real benchmark.
• Two live runs against DeepDelve's actual engine, different queries, identical failure: run 1, the same "Cretaceous–Paleogene boundary" query used to verify the delegation-cap/supersede fixes above (1321s, Report: NOT WRITTEN); run 2, a deliberately narrow single-fact query ("what is the speed of light," 934s, Report: NOT WRITTEN) chosen specifically to rule out query complexity as a confound. Both runs ended identically: "Task forcefully aborted: Agent trapped in loop. Quota exceeded multiple times for write_todos."
• Root cause, confirmed from the real session log's actual tool-call arguments (not guessed): the model calls write_todos with BYTE-FOR-BYTE IDENTICAL arguments repeatedly (the same fully-[x]-checked todo list, verbatim, call after call) — not incremental revision, a genuine repetition loop. It continues even AFTER the tool starts returning "Error: Quota reached... You MUST summarize what you've done and state clearly that you had to stop due to quota limits" — the model ignores this explicit corrective instruction and calls write_todos again with the identical content. All research work was actually done (todos fully checked, real sources fetched — 17 and 10 respectively, 0 web-search failures both runs) but the model never transitions to writing the final report; DeepDelve's own anti-loop safety net (quota + forced abort) is what stopped the run, not the model recognizing completion.
• Meets the Model Evaluation Standard's point 4 (a discard claim needs more than one run): two independent runs, different queries (one deliberately simple to rule out complexity as a confound), identical failure signature.
• Distinct failure mode from every other disqualified candidate documented so far — not the thin_coverage Planner non-convergence pattern (qwen3:4b/qwen3:8b), not citation fabrication (InternScience/Agents-A1-4B, the fine-tuned qwen3:4b), not a writer-role tool-call failure (Bonsai-8B, qwen2.5:3b-instruct). This is a genuine inability to break out of a literal identical-tool-call repetition loop even when directly instructed via a tool-error message to stop — the most severe agentic-reliability failure documented in this project's bake-off so far.
• Not a serving-layer bug — nothing here resembles the Qwen3/mistral-nemo-class Ollama passthrough bugs (point 3 of the standard): nothink and tool-calling were both independently confirmed clean at the raw API level beforehand, and the failure only appears once the model is actually driving the full multi-turn agentic loop.
• Closed out: ~/.deepdelve/config.yaml restored to the gpt-oss baseline (openai_model: deepdelve-gpt-oss:latest). Both Ollama tags for this candidate deleted (deepdelve-gemma4-12b:latest and the raw hf.co/yuxinlu1/...:Q4_K_M pull it was derived from, ~7.4GB reclaimed — they shared blobs, not 2×7.4GB), same cleanup convention as every other disqualified candidate's checkpoint (e.g. qwen3:8b's vLLM checkpoint deletion above). No trace of this candidate left on disk or in ollama list.

2026-07-26: Planner redelegation-loop fix + two new production bugs found by live testing

Addressed 3 of the "still open" items from the 2026-07-24 session (excluded: the deferred Ternary-Bonsai-27B/Vulkan fork work). All committed on main:

• 999a254 — Root-caused why the Planner renames the same research angle across retries instead of redispatching under the same task_name (e.g. background_heuristic_algorithms → _refined → _final; recurred as 43 calls for one angle under --resume-run). Cause: check_thin_coverage/check_uneven_task_investment's own directives said "redelegate ... phrased differently," which a model conflates with inventing a new task_name — nothing anywhere said otherwise. Fixed the directive wording, and added a non-blocking difflib-based similarity nudge in delegate_tasks (_looks_like_renamed_task, src/engine/orchestrator.py) that flags a likely rename against the run's own prior dispatches. Live-confirmed firing 9/20 then 4/6 real delegate_tasks calls across two separate live runs — reduces but doesn't eliminate renaming (a prompt nudge alone can't guarantee compliance). Also live-confirmed (same runs) that check_untracked_delegation's once-per-run cap (shipped 2026-07-24) holds under real pressure, and that check_report_underuses_findings fires live for the first time ever (previously unit-tested only).
• 1ed5de7 — A live smoke test hit a more severe case of the empty-response problem than the 2026-07-24 fix assumed: FindingsWriter produced nothing usable on BOTH the original dispatch AND its immediate retry, six consecutive completion-check attempts in one run, exhausting the budget with findings.md never written despite 61 real findings existing the whole time. Added a deterministic (non-LLM) salvage path: _dispatch_writer_review_fix now accepts a deterministic_fallback, wired only from the FindingsWriter call site as _build_findings_source_material's own real evidence text — written directly as findings.md when both attempts fail, instead of losing the cycle. Found and fixed a second, unrelated bug while verifying this against the real failing run's actual data (not synthetic tests): extract_cited_urls (src/utils/grounding.py) didn't stop at a trailing backtick, so the common `URL` inline-code citation style broke every fetched-URL match — would have defeated the new fallback on the exact content it exists to rescue.
• 554cfc2 — Caught by actually reading a live run's final_report.md content (not just confirming completion checks fired mechanically, per the standing project practice): a balanced 2-facet query (green tea + Roman Empire, both genuinely "covered" per RunState.coverage()) produced a final_report.md that was 100% about one facet — the other vanished entirely at the FindingsWriter consolidation stage, and NOTHING caught it (check_thin_coverage saw balanced research volume; check_report_underuses_findings compares the report against findings.md, which had already lost the data). New check_findings_underuses_evidence (src/engine/completion.py, right after check_stale_findings) compares findings.md's cited URLs against RunState's real per-task research record and fires when a covered task has ZERO of its real URLs represented. Replayed directly against the real failing run's data to confirm it would have caught the omission. Not yet observed firing organically on a fresh live run.
• 34cb72c — Added a 6th GRPO reward-function dimension, findings_underuses_evidence_response_reward (finetune/reward.py), mirroring the check above, for if/when fine-tuning resumes. Inert prep only (self-tested, not wired into train_combined_grpo.py's training loop, no training launched) — that would need per-task URL grouping threaded through the synthetic training-data generation pipeline too, a separate, bigger change.

Standing lesson reinforced this session: confirming a completion-check fired via _run_state.json proves the mechanism ran, not that the report is actually good — read the real artifact content too before calling a run's outcome confirmed.

Later the same day — a fresh live smoke test (2008 financial crisis + mRNA vaccines, --depth standard) confirmed the deterministic FindingsWriter salvage (1ed5de7 above) firing organically for the first time, live, outside any replay/unit test. But the run still failed overall (findings_ungrounded, no report written) for a separate, real reason: FindingsWriter repeatedly cited a fabricated near-duplicate insidetx.com URL across 7 independent dispatches, despite the grounding pipeline already embedding a [SYSTEM VERIFICATION WARNING: ...] directly into that finding's own summary text at generation time — the exact same shape already logged 2026-07-24 with a calendarr.com URL, now confirmed to recur with a different domain. Researched properly before fixing (RESEARCH.md §10): embedded negative-instruction warnings are documented as fragile by mechanism (naming forbidden content can prime its reproduction — the "ironic rebound" effect, arXiv:2511.12381; negation-following degrades in small models, arXiv:2601.21433), and CRAG/Self-RAG (primary papers, arXiv:2401.15884/arXiv:2310.11511) both structurally filter flagged evidence out before the generator sees it rather than annotate-and-hope.

• _is_citable_finding (src/engine/completion.py) — the shared predicate feeding _build_findings_source_material, _uncited_task_names, and _find_propagated_bad_content — now excludes findings carrying a [SYSTEM VERIFICATION WARNING marker, exactly mirroring its existing [SYSTEM RELEVANCE WARNING exclusion (2026-07-21). This deliberately REVERSES a 2026-07-22 design decision that explicitly chose NOT to exclude verification-flagged findings, reasoning they "may still coexist with other real, usable content" — a bet the live recurrence above shows doesn't hold. Since run_completion_check computes _build_findings_source_material(run_state) exactly once and reuses the identical string as both the FindingsWriter dispatch instructions AND the deterministic-salvage fallback (1ed5de7 above), this one change fixes both the recurring-fabrication problem AND the "salvage inherits poisoned evidence" problem simultaneously — no separate patch needed for either path. _should_cache_finding (src/engine/orchestrator.py, the RAG-cache write gate) already treated this warning as disqualifying, so _is_citable_finding was the odd one out relative to existing project precedent, not inventing a new policy. Verdict-matrix-adjacent unit test flipped (test_structural_checks.py) and replayed directly against the real 2026-07-26 failing run's actual _run_state.json: confirmed the fabricated insidetx.com citations no longer appear in the reassembled evidence post-fix. Tests pass, ruff clean. Explicitly did NOT build a cross-attempt confirmed-bad-URL blocklist (CiteGuard-style, RESEARCH.md §10's secondary recommendation) — the two live incidents fabricate different URLs each retry, not byte-identical repeats, so the exclusion fix above already covers both flagged problems without the added persisted-state/resume-carryover machinery a blocklist would need; kept as the fallback if this fix proves insufficient on a future run.
• Two further research-flagged ideas (VeriCite-style NLI-based evidence filtering at assembly-time, VERIMAP-style per-subtask verification functions) were deliberately NOT built this session — both real, well-scoped future directions, written up in Pending below rather than folded into this fix's diff.

Live-confirmed same day, on a fresh run (CRISPR-Cas9 mechanisms + 1929 Wall Street Crash, --depth standard): attempt 1 flagged a fabricated blinkist.com citation via findings_ungrounded — and, unlike the pre-fix insidetx.com run (a 7-attempt loop on the exact same bug), it never recurred in either findings.md or final_report.md (confirmed by direct grep: zero occurrences in both). FindingsWriter converged normally on the next attempt instead of looping. The run's eventual outcome (report_underuses_findings, then max_run_minutes mid-retry) was a separate, pre-existing, correctly-handled failure mode — the check fired accurately, the run just ran out of wall-clock budget incorporating more sources — not a regression or a new bug.

Later the same day — went through the two research-flagged Pending ideas above. The VERIMAP-style per-subtask verification-function reframing stays deferred as a real future architectural direction (unchanged, still in Pending). The NLI-based evidence filtering item turned out to be based on an incorrect assumption, caught by re-reading the actual dispatch code instead of trusting the earlier note: _run_single_task (src/engine/orchestrator.py) already runs real_grounding_problem — the FULL grounding pipeline, including NLI contradiction detection via nli_unsupported_problem — for Tier-2 Searcher dispatches. The real, narrower gap was that Tier-3 Analyzer-leaf dispatches (DocumentAnalyzer/DataAnalyzer) never got this at all — only a narrow reconstructed-URL check added 2026-07-19 for one specific bug, with no stub detection, no quote-fidelity, no content-level check, no NLI. Analyzer findings (reading and summarizing a single fetched document — exactly the shape most likely to misreport a figure or misattribute a stat) had materially weaker grounding coverage than Searcher findings, for no principled reason.

Fixed: added the same real_grounding_problem(final_text) call already proven via the Searcher-tier check to the Analyzer-leaf branch too (src/engine/orchestrator.py), alongside (not replacing) the existing reconstructed-URL check — the two catch genuinely different things (reconstructed-URL catches "cited a different-but-real URL, not the one you were given"; real_grounding_problem catches "URL matches, but the claim doesn't say what the source says"). Both append to the same verification_warnings accumulator using the identical [SYSTEM VERIFICATION WARNING marker, which _is_citable_finding already excludes structurally (this session's earlier fix) — no changes needed to the exclusion mechanism itself. Zero new machinery: real_grounding_problem/nli_unsupported_problem/the lazy CPU-only NLI singleton all already existed and were already proven in production via the Searcher-tier check. Ladder rung 2 (reuse, don't rebuild) — the original Pending note's "needs new claim/source-window extraction" premise was simply wrong. Tests pass (no new unit test added, consistent with the reconstructed-URL check's own precedent of relying on live verification for dispatch-loop wiring plus real_grounding_problem's own already-thorough existing test coverage for the logic itself), ruff clean.

Live-confirmed same day on a fresh run (Pfizer/Moderna vaccine efficacy statistics + Irish Potato Famine causes, --depth standard): two Analyzer-tier findings got flagged with problem types (claim_unsupported, stub_source) that were structurally impossible before this change — the old Analyzer-leaf check could only ever produce a reconstructed-URL message. Confirms the new real_grounding_problem call is genuinely executing for Analyzer dispatches now, not just present in the diff. nli_unsupported specifically did not fire this run (an earlier pipeline stage caught the problem first, since real_grounding_problem short-circuits on the first hit) — the wiring is confirmed correct, NLI itself being the deciding factor is still an open, narrower confirmation gap.

VERIMAP Phase 1 — a structural per-task verification ledger, same day, after scoping in a dedicated plan. RESEARCH.md §9 flagged VERIMAP (EACL 2026, arXiv:2510.17109) — a planner encodes an explicit verification function per subtask, executed by a separate Verifier before a Coordinator proceeds. Confirmed via exploration that DeepDelve's 23-check pipeline produces exactly ONE Verdict per attempt for the WHOLE run — no per-task pass/fail record existed anywhere, and several bugs fixed this session (the starvation guard, check_uneven_task_investment, check_findings_underuses_evidence) are all individually-patched symptoms of that same missing dimension. Design tension identified and resolved before building anything: VERIMAP's own mechanism has the PLANNER author each verification function — directly conflicting with RunState.coverage()'s own documented principle that DeepDelve avoids handing local models new structured conventions to follow, precisely because they've repeatedly proven unreliable at it. Reconciled by having the ENGINE compute the ledger structurally from already-existing ground truth (_is_citable_finding) instead — keeping VERIMAP's real contribution (a task has its own checkable verification state) while dropping the part of its mechanism that doesn't fit DeepDelve's own hard-won lesson.

• New run_state.data["task_verification"] ledger, keyed by task_name ({"status": "verified"|"flagged", "reason", "checked_at"}), recomputed fresh every completion- check attempt by _update_task_verification (src/engine/completion.py) from findings + _is_citable_finding — purely additive, no new Planner-facing field, no new prompt convention. A task with no findings yet is left out of the ledger entirely (still pending, not a problem).
• New check_task_verification_flagged (COMPLETION_CHECKS, right after check_thin_coverage) — the first genuinely task-scoped check in the pipeline: fires when a task's EVERY finding was excluded by _is_citable_finding, naming that SPECIFIC task in its directive rather than nudging the whole run generically. Distinct from check_thin_coverage (zero findings at all) and check_uneven_task_investment (uneven counts across covered tasks) — this catches "produced findings, but every one turned out fabricated/off-topic/contradicted." settings.task_verification_check (config_template.yaml), default enabled.
• ARCHITECTURE.md's resume-carryover allowlist updated in both src/engine/tui.py locations (_resume_run and run_cli), per its own checklist for a new persisted RunState.data key.
• Explicitly deferred (Phase 2, see Pending): actually independent per-task redispatch that bypasses the Planner's own turn — this session's ledger still only produces one whole-run Verdict per attempt, same shape as every other check; the deeper dispatch-loop rework is real, separate scoping work, worth doing only after seeing how often "flagged" recurs on real runs.

Tests: new _update_task_verification/check_task_verification_flagged scenario, new verdict- matrix row, resume-carryover extended for both TUI and run_cli copies (mirroring findings_written_citable_count's own precedent exactly). Tests pass, ruff clean.

Findings from live testing (informational, not yet acted on)

• SOTA literature review, durable conclusions merged 2026-07-20 (full detail, primary-source citations, and still-open leads in RESEARCH.md, which stays the standalone working document).

  • MAST's 14-mode failure taxonomy (arXiv:2503.13657, NeurIPS 2025) maps closely onto this project's own bug catalog, confirming DeepDelve's failures are named, published patterns rather than idiosyncratic bugs: FM 2.6 "Reasoning-Action Mismatch" = the "narrate instead of write" bug; FM 1.5 "Unaware of Termination Conditions" = the over-research/STOP-EARLY problem; FM 3.2/3.3 "No/Incorrect Verification" = the entire reason the grounding-check layer exists; FM 1.1 "Disobey Task Specification" = the exclusion-enforcement bug class. A follow-on production-telemetry replication (639K steps/23.6K runs, one closed-alpha platform) found verification gaps dominate real deployment failures while coordination failures nearly vanish (1.14% of runs) — closer to DeepDelve's own lived experience than MAST's benchmark-derived aggregate, though caveated as one platform, not peer-reviewed. A large-scale coding-agent study (arXiv:2605.29442, 16,118 validated episodes) independently found the same two DeepDelve patterns (inaccurate self-reporting ≈ "narrate instead of write"; constraint violation ≈ exclusion-enforcement) in a totally different agent domain — real, cross-domain corroboration, not a DeepDelve-specific quirk.
  • Three independent sources now converge on "verification/architecture amplifies a capable model, it doesn't rescue an incapable one": the capacity-floor paper (arXiv:2601.16280, 14B "minimum viable" for tool invocation), PIVOT (arXiv:2605.11225, "repair quality remains bounded by the underlying model reasoning capacity"), and ATLAS/AdaMAST (its own 8pp residual gap on OlympiadBench, attributed to an "architectural-vs-parametric distinction"). Relevant to every future decision about fixing a small-model gap with more structure vs. a bigger/better model.
  • A third, distinct candidate mechanism for the recurring "real fetched content silently vanishes during final synthesis" pattern (already independently observed 3 times in this project — quota-starvation drop, heterogeneous-tiering drop, citation-truncation drop, each fixed individually; see the scattered incidents at lines ~481, ~689, ~966, ~1440, ~1858 above). "Lost in the Middle" (arXiv:2307.03172, TACL 2024, foundational/highly-credible) shows models use context well at the start/end and poorly in the middle — a candidate SECOND cause distinct from truncation, not yet checked against DeepDelve's own findings-ordering. PIVOT (arXiv:2605.11225) adds a candidate THIRD: 100% of its tested models' thinking tokens fire on the FIRST turn (task decomposition), 99.2% of final-synthesis steps get ZERO thinking tokens, REGARDLESS of how large the thinking budget is raised — models don't naturally allocate reasoning to synthesis/verification, only to planning. None of these three are confirmed as DeepDelve's own root cause; each is a real, externally-sourced, testable hypothesis for the still-open "common structural cause" investigation already flagged in this file.
  • CONFIRMED, 2026-07-21 — a 4th, DeepDelve-internal mechanism, ground-truthed against real _run_state.json files on disk, not a literature hypothesis. Directly answers the heterogeneous-tiering incident's own open question above ("no quota exhaustion is visible in this run's own attempt log") — the missing variable was never a quota at all.
    • Mechanism: sub_agent_timeout_minutes (src/engine/orchestrator.py:701-703,721-738) is a hard wall-clock cutoff on a sub-agent dispatch's ENTIRE stream, independent of the quota pools and invisible to completion_check_attempts' logging. It can fire AFTER a Searcher/WebSearcher has already fetched a real URL (via fetch_url_to_workspace, tracked in task_fetched_urls_ctx) but BEFORE the model emits its synthesized summary text — leaving a run_state.findings entry with a REAL source_url whose entire summary is verbatim "[SYSTEM: task '<name>' cut short -- sub_agent_timeout_minutes (N) exceeded...]", zero actual content. When check_thin_coverage (or a completion-check retry) redispatches the SAME task_name — a fresh-context dispatch with no memory of the earlier partial progress — the retry sometimes narrates genuinely good synthesized content but without re-fetching a URL this round (relying on reasoning/recall, or citing something not captured as a real fetch), landing in add_finding's task_name-fallback bucket (see the source_url fabrication fix earlier this file). The real content and its real URL end up split across two separate, un-mergeable findings entries for the same task — one has the citable URL with nothing worth citing, the other has real content FindingsWriter cannot legitimately cite (per its own "never a bare task name" rule, correctly enforced). Net effect: the whole topic silently vanishes from findings.md, with nothing in the attempt log pointing at why.
    • Confirmed NOT a one-off, ground-truthed against every real run on disk (107 _run_state.json files under research_output/): 44 findings entries across 9 distinct runs show the exact real-URL + timeout-cutoff-only-summary pattern. In 3 of those 9 runs, the full split-brain pattern is present — a same-task_name entry with substantial real-looking content (300+ chars) under the task-name-fallback source_url — confirmed reproduced across THREE different dates and backends: 20260714_201217 (top_heuristics), 20260718_141225 (the exact heterogeneous-tiering A/B run analyzed above — background_heuristics's two cutoff entries carry the real sciencedirect.com/forecastio.ai URLs with nothing but the cutoff message, while a THIRD background_heuristics entry has a genuinely detailed N-BEATS/ TFT/arXiv writeup under the bare task-name fallback), and 20260721_174718 (this same session's killed qwen3:8b vLLM re-test run).
    • Candidate fixes, not yet implemented, pending user sign-off (touches the shared dispatch loop, every sub-agent's blast radius):
      1. Ring-fence the timeout, mirroring the existing quota-fairness fix. check_quota already ring-fences remaining QUOTA for a task that's shown real fetch activity (see the quota-starvation incident above); the same philosophy applied to sub_agent_timeout_minutes (extend the deadline once, by a bounded amount, for a task that has a real task_fetched_urls_ctx entry but hasn't finished synthesizing) would prevent the cutoff from ever severing a real fetch from its own summary in the first place — closest to a true root-cause fix, most consistent with prior art in this codebase.
      2. Give a retry context of its own prior partial progress. A redispatch of a task_name that already has an on-disk fetched file from an earlier cut-short attempt currently starts from a totally blank slate. Telling it explicitly ("you already fetched <url>, saved as <filename> — delegate it to an Analyzer now instead of searching again") would let the retry finish the SAME real work instead of restarting and orphaning it.
      3. Contain, don't recover: at minimum, a findings entry whose entire summary is the cutoff system message should be excluded from _build_findings_source_material's citable entries (same treatment as today's non-http fallback fix) and named in the uncited-tasks note instead — turns a silent loss into an explicit, acknowledged gap even without recovering the content. Smallest fix, but doesn't rescue anything.
    • 1+3 IMPLEMENTED, 2026-07-21, after a dedicated pre-implementation audit against the real code and external prior art (not just re-reading the plan). Audit found the recommendation itself sound but surfaced one real, non-obvious implementation risk before it shipped: an unbounded deadline extension could push a dispatch's task_deadline past _build_client's own sdk_timeout (floored at 3600s), reintroducing the exact "SDK's blunt timeout wins the race" bug that timeout's own comment already documents fixing once. Reference check: Temporal's documented pattern for this exact failure class (long LLM activity, timeout severs partial progress) is heartbeat-and-resume — record partial progress as a heartbeat, extend the deadline instead of killing the activity — structurally identical to fix 1, confirming it's the standard answer, not a bespoke hack. Separately checked whether "just add real checkpointing" (candidate fix 4, previously deprioritized) has become easier anywhere since — no: LangGraph, the most checkpoint-mature agent framework, still has no native mid-node partial-state persistence in 2026 (its own forum confirms the workaround needs a raw async generator with every field manually wired to reducers, "fragile and easy to misconfigure"), confirming fix 4 was correctly scoped as new capability, not underestimated.
      • Fix 1 (src/engine/orchestrator.py): _ring_fenced_deadline (new pure function, ~line 171, next to _extract_excluded_topics) computes the capped extension — min(task_start + sub_agent_timeout_minutes*2*60, task_start + sdk_timeout_ceiling-60) — so the fix can never reintroduce the SDK-race bug. _sdk_timeout_ceiling_seconds (new, in create_local_agent, mirrors _build_client's own sdk_timeout formula exactly) is the cap. _try_extend_deadline_once (new closure in _run_single_task, next to task_deadline) checks task_fetched_urls_ctx and extends once per dispatch; wired into both cutoff sites (the remaining <= 0 branch and the asyncio.TimeoutError except, ~line 900-920) via continue instead of an immediate cutoff.
      • Fix 3 (src/engine/completion.py): _CUTOFF_ONLY_SUMMARY_RE (new, right before _build_findings_source_material) matches the cutoff marker text when it's the ENTIRE summary (not when real content precedes it — a partial synthesis still gets shown). A matching entry now falls into the existing uncited_task_names branch instead of being rendered as a citable ### Source: entry.
      • Both covered by test_structural_checks.py (new assertions: _ring_fenced_deadline's cap math including the case where the SDK ceiling leaves no room to extend at all; _CUTOFF_ONLY_SUMMARY_RE against both marker variants and against real-content-plus-marker summaries) — full suite still passes.
  • Comparative survey against 5 other real deep-research-agent projects (Tongyi DeepResearch, dzhng/deep-research, CYC2002tommy/Deep-Research-Agent, SkyworkAI/DeepResearchAgent, nashsu/ llm_wiki — all already credited in README's References) answered a deliberate test question from the user honestly: DeepDelve's 10-layer grounding pipeline is more elaborate than any of the 5 for the SPECIFIC problem of post-hoc citation verification on a small/local model — but this is explicitly NOT "most sophisticated deep research agent, period." Tongyi DeepResearch solves reliability via a much larger purpose-trained model, a different and likely more effective lever DeepDelve's own local-only constraint doesn't have access to; and "sophisticated mechanism" is not the same claim as "proven real-world catch rate" — most of DeepDelve's own grounding checks still lack real-captured-fabrication test coverage (see "Test coverage debt" note in session history). See RESEARCH.md §7 for the full, appropriately-bounded writeup.
  • A non-generative routing-classifier design for delegate_tasks is now a scoped "Pending" item (see above) rather than a research note, prerequisite data already confirmed sufficient.

• Full grounding/completion-check compliance audit (2026-07-18), all 12 README-claimed guarantees re-verified against the actual code, not just the docs. Checked each of: URL grounding with path-boundary matching, content-level zero-fact-overlap, non-URL citation detection, regulation- identifier check, stub-fetch detection, uncited_claims, NLI entailment (nli-deberta-v3-small), atomic-claim segmentation (decompose_claim_segments), FEVER-style cross-source contradiction, topical relevance (bge-reranker-v2-m3), coverage accounting (RunState.coverage()), and the test_structural_checks.py verdict-matrix pin. All 12 found genuinely implemented and reachable from the real completion-check flow (GROUNDING_CHECKS/ COMPLETION_CHECKS in src/engine/completion.py:563-582) — no dead code, no orphaned function, no early-return that silently skips a check, no always-false gating condition. Every check fails open (returns None) on model-load failure rather than crashing a run, confirmed as deliberate documented behavior rather than an oversight. check_not_grounded's ordering as the last, generic catch-all in GROUNDING_CHECKS is deliberate so the more specific verdicts fire first. Net: the README's grounding-guarantees section does not overclaim relative to the code as of this date.

• Grounding check verifies provenance, not topical relevance. A live GOA (Grasshopper Optimization Algorithm) research query got a citation from globaldrivetozero.org — actually fetched, and sharing surface terms like "GOA"/"Goa" — that's actually about the Indian state of Goa's EV policy, not the algorithm. The URL-presence + term-overlap check passed it because it only checks "was this fetched" and "do terms overlap," not "is this source about the same subject." Acronym collisions are the clearest way to trigger this; unclear how common the failure mode is outside them. Fixed 2026-07-14 — see "Completed" (Phase 4, topical_relevance_problem).

• JS-gated pages return bot-challenge stubs, not content. Several fetches (Cloudflare "Just a moment...", a "Human Verification" page, a Prezi slide deck) came back as 16-18 byte stubs since the fetcher doesn't execute JavaScript. (Fixed for most cases — see "Completed": headless/headed-browser fetch fallback, 2026-07-14. Recovers Springer (headless-sufficient) and MDPI (needed headed). NOT a universal fix: a genuine Cloudflare Turnstile challenge (ScienceDirect) resists both headless AND headed Chromium regardless of patience or navigator.webdriver spoofing — confirmed to be automation/CDP-fingerprint detection, not a solvable timing issue, and deliberately not pursued further; see the ScienceDirect sub-bullet above for the full investigation. Still correctly falls through to the stub flag rather than silently failing.)

• A citation being present in a report's "Sources" list doesn't mean it was fetched. Across several market-research runs, more than half of named sources were routinely never actually fetched (recalled from the model's training data) — and when independently fact-checked, specific statistics tied to unfetched sources were measurably wrong, usually understated.

• Hard exclusion rules ("do not research sector X") repeatedly fail to hold, confirmed across at least 2 independent runs with different prompt wordings: an explicitly-excluded "Agricultural"/"agribusiness" sector got researched and included in the final report anyway — once purely from memory, once with the model actually delegating and fetching a real source for the excluded sector. Simply naming the exclusion in the prompt isn't enough; delegate_tasks's existing dispatch-time skip (_extract_excluded_topics) only stopped NEW research on the topic, not the topic showing up in the final report anyway via a sibling task's tangential findings. Fixed 2026-07-14 — see "Completed" below (check_excluded_topic).

• Non-URL "citations" evade the grounding check entirely. A live report sourced several claims to "Expert opinion from a cold storage facility manager in Colombia" — not URL-shaped, so extract_cited_urls never sees it, even though it's exactly as ungrounded as a fabricated URL. The grounding check's whole model is "cross-reference cited URLs against fetched URLs" — a citation with no URL at all currently gets a free pass. Fixed — see "Completed" above (non_url_citation_check).

• Scaling down scope (12 sectors → 5) improved surface polish, not actual grounding rate. A 5-sector re-run produced far more plausible-looking, consistently-formatted citations than a 12-sector run, but cross-referencing against _run_state.json's real fetched_urls showed most of them were still fabricated — only 5 URLs were ever fetched all run, while the final report cited well over twice that many distinct domains. Fewer sectors did not proportionally reduce the fabrication rate.

• Shared cumulative web_search quota pool can starve a specific task of the ability to synthesize what it already fetched (2026-07-14). Live sales-forecasting benchmark run (research_output/i_want_documentation_on_heuristic_algoritms_for_de_20260714_225720/): the final report was well-grounded on its technical content but silently dropped the Colombia cultural-context section (holidays/paydays) ENTIRELY, despite the query explicitly requiring it and the research genuinely happening — NOT the same bug as the FindingsWriter dedup fix shipped earlier this session (that fix worked correctly here; the empty-summary entry reached FindingsWriter's material intact, there was just nothing usable in it). SubAgent_Colombian cultural events affecting sales was dispatched 4 separate times across the run's retries. Dispatch #1 genuinely fetched 2 real sources (timeanddate.com/holidays/colombia/2024, an ADP payroll-calendar article) but its RunState.add_finding entries have EMPTY summaries — it fetched but never got to actually analyze/synthesize before being cut off. Dispatch #4 (the last one) has a real summary, but it's just an apology: "I've reached the maximum number of web-search calls allowed for this session (15). No sources were successfully fetched..." Root cause: web_search's quota (build_quota_pool) is ONE shared, cumulative pool across every sub-agent in the run — other tasks (particularly "Top 5 common heuristic algorithms," which shows heavy repeated web activity in this run's findings) burned through the pool first, so by the time the Colombia task got redispatched on retries #3/#4, the shared quota was already exhausted, and it could never finish analyzing the sources it originally fetched. Partially fixed 2026-07-18 — see "Completed" above (check_quota's ring-fence), which addresses angle (b) below (a dispatch that already fetched something real no longer gets hard-blocked mid-synthesis). Angles (a)/(c) remain open — candidate angles: (a) a per-task reserved minimum quota allotment, (b) [addressed] protecting/ring-fencing a task's remaining quota once it's shown real fetch activity (distinguishing "genuinely progressing but interrupted" from "never started"), (c) some kind of fairness/round-robin ordering across redispatched tasks instead of first-come-first-served on a shared pool. Distinct from retry_quota_topup (which already tops up the pool between completion-check ROUNDS) — this is about fairness WITHIN a round, across concurrently/sequentially dispatched sibling tasks sharing the same pool.

• gpt-oss hallucinates entire tool names, not just filenames (2026-07-12). Distinct from the fuzzy-filename problem fixed this session (a real tool called with a garbled argument) — this is the model inventing a function that was never in its schema at all: grep_search? and justify both fired as literal function-call names in one live run (heuristic-algorithms sales-forecasting query), 3 occurrences total. Each one only cost a turn (clean error, malformed_tool_call_nudge path, sub-agent recovered without stalling) but three in a single run is a real pattern worth its own investigation, not noise to fold into the filename fix.

  • Investigated 2026-07-14 — no code fix, re-tested live, existing infra already covers it. Re-ran the EXACT same benchmark query live (research_output/i_want_documentation_on_heuristic_ algoritms_for_de_20260714_225720/, 939.3s, clean pass, converged by attempt 3): zero hallucinated-tool-name errors this time, out of 11 total tool errors recorded (all legitimate — a real missing-field validation error, a real missing file, expected quota-exhaustion messages). Doesn't prove the underlying tendency is gone (one run against one prior run is weak evidence either way — could be genuine improvement from the many structural fixes shipped since 2026-07-12, or just run-to-run variance), but two things make further code investment unjustified without stronger recurrence evidence: (1) the tool schema the model sees is the real, structural OpenAI-style function-calling schema (name/description/params passed via the API's own tools parameter), not prose — occasional hallucination despite having the correct schema in context is a generation-sampling failure, not a missing-information one, so "tighter prompt framing" was never likely to help; (2) tool_result_error_nudge (src/engine/orchestrator.py:268, shipped 2026-07-14, AFTER this finding was first recorded) already generically pattern-matches the exact Requested function "..." not found error text and gives a corrective nudge — so even if this recurs, it's no longer a silently-wasted turn, it costs at most one extra turn with real guidance, same fix that already closed the "zero recovery path" gap for this exact error class. Revisit only if this resurfaces with real frequency data across multiple runs, not as a standalone investment.

• gpt-oss endgame-collapse reproduced again, fresh data point (2026-07-12), now also observed INSIDE Builder (2026-07-13). Same live run above: 9 completion-check attempts, cascading web_search/grep_workspace_file/fetch_url_to_workspace quota exhaustion across multiple re-delegation rounds (including a genuine QuotaAbortException nested-agent abort), before finally falling back to the quarantine-restore path at attempt 9/9 — the query (peer-reviewed sourcing for heuristic algorithms + deep learning + multi-franchise sales forecasting, a 3-way AND) never had a real source satisfying all three criteria. Already tracked as a known gap (runs 11/13) — not a new finding on its own, but confirms it's not resolved and reproduces on a genuinely hard query, not just a fluke. Re-tested 2026-07-13 against the same exact query after the Builder architecture shipped: the collapse shape moved, it didn't disappear — Build→Review→Fix correctly fired 3 times on real not_grounded problems, but on attempts 4-6 Builder itself ran out of the shared write_workspace_file quota and fell back to narrating the report as chat text instead of writing it (Builder's own output: "I'm unable to create new files because the write_workspace_file quota has been exhausted") — the identical failure shape the Planner used to exhibit, now happening one level down. The quarantine-restore fallback still worked exactly as designed both times: final artifact carries a loud warning banner (or is fully restored from the best surviving quarantined draft) instead of a fabricated clean-looking report or a lost one. See "Pending" below for the quota-sharing angle this surfaced.

• Line-scoped claim grounding (2026-07-12): claim_grounding_problem compared WHOLE-report terms against each source, so generic shared terms masked per-claim fabrication (run 12's flagship figure was absent from its cited source but passed via other lines' overlap). Now each line with a fetched citation is checked against its own source(s) — the regulation-check pattern generalized; conservative as before (≥1 checkable term + zero overlap only, URL slugs stripped).

• Structural eval scorer (2026-07-12): new eval_type: structural in eval/evaluate.py — rubric tier 1 scored deterministically from _run_state.json + workspace files (cited⊆fetched, findings.md grounded, no salvage/quarantine banner, no unresolved final problem), which no other scorer read at all.

• Four concrete findings from a fresh live run of the standing sales-forecasting benchmark (2026-07-13, later the same day the Builder loop shipped) — user killed the run after it stalled; each finding traced to an exact file/line, not guessed:

  • _strip_trailing_punct (src/utils/grounding.py:59-66) didn't strip a trailing *. (Fixed 2026-07-14.) Builder's own citation format **[Title](URL)** puts ** immediately after the link's closing ) with no space; the existing unbalanced-)-stripping loop only fired when the string ends with ), so a URL ending in )** was never cleaned up. Confirmed live: two of this run's four completion-check attempts were not_grounded verdicts citing the literal string ...546e2a498c2f)** as "unverified" — a genuinely-fetched, correctly-cited source false-flagged as hallucinated purely by this string-handling gap, burning half the run's retry budget on a checker bug, not a model failure. Fix: added * to the initial rstrip() char set, stripped BEFORE the balanced-paren check so a bold-wrapped URL's real trailing ) is exposed to it correctly (verified against a bold URL that also has its own internal balanced parens, e.g. a Wikipedia disambiguator page — both layers now resolve in the right order). Two new assertions in test_structural_checks.py.
  • Sub-agent "tool not found"/"argument parsing failed" errors had zero recovery path. (Fixed 2026-07-14.) Confirmed via code trace: these come back from agent_framework's SDK as in-band tool-result text, never as exceptions, so they never reached _run_single_task's except block and never triggered the existing malformed_tool_call_nudge (which only covers transport-level "error parsing tool call" failures). Confirmed live: a SubAgent_BuilderFix retry hallucinated a call to delegate_tasks (Builder's real tool list never includes it — the model invented the call, not a config leak); a separate sub-agent called a malformed grep_workspace?; PeerReviewer tried reading a nonexistent workspace.txt. Each burned a turn with no corrective nudge of any kind, unlike the Planner's own conversation. Fix: new engine/orchestrator.py::tool_result_error_nudge, a sibling of malformed_tool_call_nudge scoped to the exact SDK error strings pulled from agent_framework/_tools.py source (not guessed) — Error: Requested function "{name}" not found. (hallucinated tool name), Error: Argument parsing failed. (rejected arguments), and tools/fs.py's Error: '{filename}' not found. (missing file). Wired into _run_single_task's stream loop: the pending nudge is overwritten on every function_result seen, so a LATER successful call after an earlier error (the model already self-correcting within the SDK's own internal turn) clears it — only an error still standing at the end of the stream gets nudged, capped at 2 retries like malformed_retries. Deliberately narrow (three specific, evidence-backed error shapes, not every possible tool failure) so a legitimate business-logic error (a real search that genuinely failed, a quota genuinely exhausted) doesn't get blindly retried when that wouldn't help — verified against both the three matching cases and two non-matching ones (a real fetch-success string, web_search's own timeout error) with no false positives. New assertions in test_structural_checks.py. Deliberately NOT extended to the Planner's own loop (run_agent/run_cli in engine/tui.py) despite this project's usual TUI/CLI parity rule — this is a reasoned scope decision, not an oversight: the Planner already has independent recovery via its multi-attempt completion-check loop (several full outer retries across an entire run, each with fresh nudges and quota top-ups), unlike a sub-agent's single one-shot dispatch with no outer safety net at all — the asymmetry this fix closes is specific to sub-agents, not a gap in the Planner too. Relationship to the researched LangGraph RetryPolicy pattern (see the earlier-recorded research-pass note): that pattern's retryable-vs-fatal split maps onto DIFFERENT layers of this codebase rather than one function — the genuinely retryable class (timeout, rate-limit, transient parse garble) is exactly what web_search's own daemon-timeout fix and the SDK's built-in 429/5xx backoff already handle; tool_result_error_nudge covers what that pattern calls fatal (hallucinated tool name, rejected arguments) — except here "fatal" doesn't mean "give up," it means "immediately actionable by telling the model exactly what's wrong," which is what the nudge does.
  • web_search/probe_search_health (src/tools/web.py) had no outer wall-clock timeout. (Fixed 2026-07-14.) DDGS() is built with no explicit timeout at either call site, relying on the ddgs library's own internal 5s-per-engine default — not a real ceiling, since ddgs runs engines in a ThreadPoolExecutor and its context-manager exit calls shutdown(wait=True), which blocks until every thread finishes regardless of the nominal per-engine timeout. Confirmed live: the process ended up blocked with one established TCP connection open 9+ minutes to a yandex.ru-resolving IP (not an intentional backend anywhere in this codebase — almost certainly a redirect inside ddgs), local model unloaded, GPU idle. Generalizes the already-tracked "no liveness/stall detection" gap (previously scoped to hosted/NIM runs only) to local web_search too. Fix: tools/web.py::_run_with_daemon_timeout — a real threading.Thread(daemon=True) with .join(timeout), not a bare asyncio.wait_for(asyncio.to_thread(...)). That distinction mattered in practice: a plain wait_for DOES unblock the awaiting coroutine on time, but its underlying executor thread is not a daemon thread, so if the search call never actually returns (confirmed against two real GitHub issues, HKUDS/nanobot#2804 and microsoft/amplifier#219, describing ddgs's primp Rust HTTP client blocking below anything asyncio can interrupt), the orphaned thread then blocks the WHOLE PROCESS from exiting cleanly at the end of a run — verified directly with a time.sleep(999)-hung call: bare wait_for/to_thread times out the caller fine but the process itself never exits; the daemon-thread version times out the caller AND lets the process exit cleanly. settings.web_search.timeout_seconds (default 20), shared by both web_search's two attempts and the pre-run probe_search_health check (src/engine/tui.py, run_cli). Process-based isolation (spawn+kill a subprocess) was considered and rejected — it would require calling ddgs from a picklable module-level worker, breaking the existing in-process ddgs.DDGS monkeypatch test in test_structural_checks.py since a subprocess re-imports fresh, unpatched modules; the daemon-thread approach closes the same gap (including the exit-hang) without that cost.
  • Sub-agent status widgets had no staleness indication. (Fixed 2026-07-14.) (src/engine/tui.py, handle_agent_update). Unlike ProcessingWidget/ToolCallWidget's animated timers, the per-sub-agent Static widget showed "▶ {agent_name} executing..." with no timer and no upper bound — if the underlying dispatch never resolved (exactly what the stall above causes), it stayed frozen on "executing" forever with zero visual signal anything was wrong. Same bug class as the already-fixed ProcessingWidget elapsed-counter issue, but that fix never got applied here — this is what "stuck agent" looked like from the user's side that night. Fix: new SubAgentStatusWidget class (mirrors ProcessingWidget's animated-dots + live elapsed-seconds pattern exactly), swapped in at the one mount site in handle_agent_update; mark_finished(elapsed) replaces the old one-shot .update(...) call on completion. Also wired into /stop's existing widget-cleanup block (alongside ToolCallWidget/ProcessingWidget/ThinkingWidget) so a manually-stopped run marks these stopped too instead of leaving them frozen mid-animation — a related gap the bare Static couldn't have supported anyway (no mark_stopped method existed to call).
  • Full prioritized fix plan (strip-punct fix → search timeout → sub-agent error nudge → widget staleness indicator) was written to a local plan file during triage. All four items fixed 2026-07-14 — see "Completed" above/below.
  • Builder's write_workspace_file quota was shared with the Planner and every prior Builder dispatch, with no guaranteed headroom of its own. (Fixed 2026-07-14.) On a long, many-retry run, the shared pool could be exhausted by the time a later corrective Builder dispatch needed it, degrading Builder to narrating the report as chat text instead of writing it — the same "narrate instead of write" failure the Planner used to be prone to, now one level down. retry_quota_topup already topped up the pool on every completion-check retry, so this wasn't starved by DEFAULT config, but a config with a low write_workspace_file limit/topup would starve Builder specifically. Fix: new engine/completion.py::_ensure_builder_write_quota_headroom, called right before every _dispatch_build_review_fix dispatch (after the existing per-attempt topup_quota_pool) — tops up ONLY write_workspace_file, and only by the exact headroom this one cycle could need (2 units: Builder's initial rewrite + one possible corrective Fix pass), not a blanket amount that would also quietly inflate the Planner's own budget. Chose this over the other option on the table (a separate Builder-reserved quota pool) because a reserved pool would work against build_quota_pool's deliberate one-shared-cumulative-pool-per-role design, not just extend it. New unit tests in test_structural_checks.py (near-exhausted pool topped up to exactly 2 headroom, a pool with plenty already left untouched — no silent inflation — and a pool missing the key entirely, no KeyError).

Model bake-off & backend investigation log (2026-07-11 through 2026-07-21)

Real, finished testing/investigation work — every entry below concluded (a model disqualified, a backend confirmed/rejected, a benchmark scored), not open backlog. Kept separate from "Completed" since most entries are investigation conclusions rather than shipped code changes; kept separate from "Pending" since none of it is still-to-do. See README's "Model choice" table for the current-state summary; this section is the full evidence trail.

• Local-model bake-off: Gemma 4 12B, Bonsai-8B, and qwen3:4b vs. gpt-oss:20b (found/verified 2026-07-13, smoke-tested and partially live-tested 2026-07-14) — two real local-model candidates surfaced by a 3-model research pass, independently verified (not taken on trust — one of the three research responses fabricated citations, see below). Gemma 4 12B (Google, Apache 2.0, released April/June 2026): dense, encoder-free multimodal, ~7.1-7.6GB at Q4_K_M GGUF (~6.7GB on the QAT Q4_0 build) — comfortably inside the 16GB ceiling. Bonsai-8B (PrismML, Apache 2.0): trained natively at 1-bit precision, 1.15GB, scores 73.3% on BFCL (format-compliance tool-calling) — beating every model PrismML tested — but drops to 43.8% on NexusRaven (semantic API understanding) vs. Qwen3.5-9B's 75%, a real and confirmed weakness on complex tool semantics, not smoothed over in the source.

  • Derived deepdelve-* tags created (FROM <base>, PARAMETER num_ctx 16384, matching the project's existing deepdelve-gpt-oss pattern) for both, plus two more candidates the user separately surfaced: granite3.1-dense:8b (IBM, Apache 2.0, 5.0GB, 128K context, model card claims function-calling) and phi4-mini:3.8b (Microsoft, 2.5GB, 128K context, model card claims function-calling) — both attractive on paper for being lightweight with a large context window. Also fixed a real hygiene issue found along the way: the SetneufPT-uploaded Gemma 4 Ollama tag ships a baked-in SYSTEM "You are a coding agent. Be concise." default (verified live it's fully overridden by DeepDelve's own system prompt at runtime, so not a functional bug — but cleaned up in deepdelve-gemma4-12b's Modelfile regardless, since the default is actively misleading for a research agent).
  • Tool-calling smoke test (2026-07-14), DeepDelve's real delegate_tasks schema (2-task nested array, task_name/instructions/agent_id), direct /v1/chat/completions calls: granite3.1-dense and phi4-mini both FAIL outright — despite each model card explicitly claiming function-calling support, and Ollama's own capability introspection listing tools, both narrated the tool call as literal text (<tool_call>[{"arguments":... / [{"type":"delegate_tasks","tasks":...) instead of emitting a real structured tool_calls response, every single attempt. Identical failure class already documented for devstral:24b in this same file — a model that narrates perfectly-formatted JSON instead of calling the tool is exactly as unusable here as one that can't format JSON at all, since DeepDelve is 100% tool-call-driven with no narration fallback. Both disqualified, pulls removed (ollama rm granite3.1-dense:8b deepdelve-granite3.1-dense phi4-mini:3.8b deepdelve-phi4-mini) — not worth carrying disk space for models that fail the first, cheapest gate. deepdelve-bonsai-8b and deepdelve-gemma4-12b both PASS — real structured tool_calls, correctly shaped 2-task array, valid task_name/agent_id on both; Gemma 4's instructions fields were notably more detailed (289-356 chars) than Bonsai's (73-102 chars), a first hint in Bonsai's favor of the NexusRaven-flagged semantic-thinness concern above, though not yet confirmed at full-benchmark scale.
  • First real end-to-end benchmark data point, Gemma 4 12B (2026-07-14): ran the standing sales-forecasting benchmark (eval/sales_forecasting_benchmark.md) live end-to-end, config pointed at SetneufPT/Gemma4-12B-IT-QAT_Q4_64K_16GB-GPU:latest. Result: Report: NOT WRITTEN after 33 minutes (1998s) — but a clean, honest failure, not a stall or a silently- accepted fabrication, and this run is what actually validated the same day's 5 reliability fixes end-to-end: web_search 26/26 calls succeeded with zero failures (the timeout fix never even needed to fire), 27 real sources fetched, the grounding check correctly rejected 4 straight ungrounded findings.md attempts, and the process exited cleanly with a clear forensic verdict instead of hanging. The actual failure was model-specific: 22 occurrences of delegate_tasks call rejected (sub-agents repeatedly submitting placeholder/pronoun-only/ cross-task-dependent instructions — the existing validator's already-detailed guidance, not a missing-nudge gap), and a visible reasoning-loop pattern near the end ("Wait, I'll just do it. (Action)", repeated ~13 times with no actual tool call) before context_budget_chars cut the turn short. Same failure shape as mistral-nemo (README "Model choice" table): passes an isolated schema smoke test, ceilings on the real multi-step benchmark.
  • Bonsai-8B benchmark result (2026-07-14): Report: NOT WRITTEN after 484.3s — DISQUALIFIED for a more severe reason than Gemma 4's. Ran the same standing sales-forecasting benchmark, config pointed at deepdelve-bonsai-8b. Research itself worked completely fine: 22 real findings recorded, 15 real sources fetched, zero web_search failures — the failure is entirely isolated to the FindingsWriter/PeerReviewer writer-tier roles. Traced through the persisted session log turn-by-turn (not just the final verdict): FindingsWriterFix_attempt1 through attempt8 each "Finished" and PeerReviewer "found no issues" each time, yet check_missing_findings kept re-firing every single retry and findings.md never existed on disk at all by the end. Root cause confirmed by reading the actual logged tool calls: FindingsWriterFix_attempt1's only event was a bare, empty text response — it never called write_workspace_file. ReviewFix_attempt1 never called read_workspace_file either — it went straight to "REVIEW: CLEAN"\n\nThe file findings.md appears to be a well-structured report... for a file it never opened and that never existed. This repeated across all 8 attempts before the retry budget exhausted. Distinct from and worse than every other failure flavor documented in this project so far (Gemma 4's reasoning loops, qwen3:4b's repeated-identical-write-calls below, gpt-oss's hallucinated tool names): those all at least attempt real tool calls; Bonsai-8B skipped tool calls entirely in a role requiring read-then-reason-then-write composition, while its simpler single-shot Searcher/Analyzer tool calls (web_search, fetch, read/grep) worked reliably throughout the same run. Also exposes a real structural gap worth considering separately: _dispatch_writer_review_fix's clean-check only string-matched "REVIEW: CLEAN" in the response text, with no verification that a read_workspace_file call actually happened first — a model confident enough to fabricate the sentinel could defeat the review entirely. This is a model-reliability finding, not a code bug, and the disqualification stands regardless. Bonsai-8B ruled out as a gpt-oss:20b replacement. Hardening fixed 2026-07-14 (src/engine/completion.py::_dispatch_writer_review_fix, commit bfd2cd5): cross-checks the read_workspace_file quota's used-count delta around the PeerReviewer dispatch — a CLEAN verdict with zero new reads is now treated as ISSUES FOUND, forcing the existing corrective Fix pass instead of being trusted. Fails open when the quota isn't tracked at all, so a config without it doesn't get every review falsely distrusted. New tests in test_structural_checks.py (_clean_check_read_verification_scenario): a fabricated CLEAN with zero reads forces the corrective pass, a CLEAN backed by a real read is still trusted.
  • qwen3:4b added as a fourth candidate (2026-07-14), specifically sought out as "Bonsai-like but more context": user asked for smaller/lighter alternatives with a bigger context window than Bonsai's 64K. Checked and rejected first: Microsoft's official BitNet b1.58-2B-4T doesn't even run on Ollama (needs Microsoft's own separate bitnet.cpp runtime, incompatible with llama.cpp) and caps around 4-8K context regardless; PrismML's own newer "Ternary Bonsai" family (1.58-bit, released 2026-04-16, same company as Bonsai-8B) turned out to be a context downgrade, not an upgrade — 4096 tokens via llama.cpp/Ollama, worse than the original 1-bit Bonsai-8B's 64K. qwen3:4b (Alibaba, Apache 2.0) is the real find: 2.5GB Q4_K_M, 262144 native context (4x Bonsai's 64K, in the same size class as the disqualified phi4-mini), established Ollama tool-calling track record in this project already (qwen2.5-coder, qwen3.6 both work). Derived tag deepdelve-qwen3-4b created (num_ctx 16384, same pattern). Passed the real delegate_tasks smoke test cleanly: real structured tool_calls, correctly shaped 2-task array, valid task_name/agent_id — and showed real semantic routing judgment at this early stage, not just format compliance: correctly sent the more academic/technical task ("hybrid statistical+DL forecasting methods") to AcademicSearcher and the cultural/retail task to WebSearcher, rather than routing both identically. Instructions detail (143-171 chars) sits between Bonsai's terse style (73-102) and Gemma 4's richer one (289-356). Not yet run through the full sales-forecasting benchmark — that's the same next step as Bonsai-8B above.
    • New reliability finding (2026-07-14, Phase 4 smoke-test session): as FindingsWriter on a trivially simple factual query ("boiling point of water at sea level"), qwen3:4b called write_workspace_file 10 times in a row with near-identical content (confirmed via the persisted session log: every call succeeded cleanly, "Wrote 'findings.md' to disk.", no error/rejection anywhere) instead of recognizing the file was already correctly written and stopping — only the existing write_workspace_file quota (10) correctly halted it, with a clear "you MUST summarize... and state you had to stop due to quota limits" message. Not a hang, not a code bug — the quota mechanism worked exactly as designed; this is a genuine qwen3:4b tool-calling non-convergence pattern, distinct in shape from Gemma4's own documented reasoning-loop tendency (repeated delegate_tasks/narration without a real tool call) and gpt-oss's hallucinated-tool-name pattern — same broader "small local model doesn't recognize task completion" failure class, third distinct flavor of it now observed across three different models in this project. Real cost: burned enough wall-clock across 2 separate live smoke-test attempts (this model, this exact query) to exceed a 15-20 min budget each time, purely on redundant write_workspace_file calls before the run ever reached its later stages. Not yet run through the full sales-forecasting benchmark, so unclear if this is systemic to qwen3:4b's FindingsWriter behavior specifically or an isolated occurrence.
    • Full sales-forecasting benchmark result (2026-07-14): inconclusive, not a verdict. Ran the same standing benchmark as Bonsai-8B/Gemma 4 above, config pointed at deepdelve-qwen3-4b. The research phase completed cleanly (Colombia-specific holidays/ paydays identified from Banco de la República, cultural cross-check against Latin American market studies, top-5 ML techniques evaluated) and the Planner correctly recognized completion and stopped delegating. The Write→Review→Fix cycle then began (FindingsWriterFix_attempt1 → ReviewFix_attempt1 flagged issues → corrective pass), but the whole process was killed by the smoke test's own 40-minute outer timeout before it could finish. Confirmed via journalctl -u ollama this was NOT a hang: right up to the kill moment, Ollama was actively, continuously decoding a response (steady ~59-62 tok/s, climbing token count, no stall) — a fairly high volume of smaller, somewhat repetitive tool calls in earlier sub-agent turns (consistent with the redundant-tool-call finding above) ate enough of the budget that the writer-tier cycle didn't have room left to converge, not that the model got stuck. Recorded as inconclusive rather than a pass or fail — user chose not to re-run with a longer cap this session; re-running with more wall-clock budget is the next concrete step before drawing any verdict on qwen3:4b vs. gpt-oss:20b. Flagged as a real data point for the eventual full bake-off comparison, not yet a disqualification.
    • Conclusive re-run (2026-07-18), no outer timeout this time: Report: NOT WRITTEN after 1214.2s (20.2 min), retry budget exhausted (8/8) on an unresolved thin_coverage verdict. qwen3:4b is DISQUALIFIED as a gpt-oss:20b replacement. Real research did happen (5 sub-agent dispatches, brave_web_search calls fired throughout), but only 1 real source ever landed (statista.com/.../music-events/colombia) against 4 delegated tasks. The disqualifying behavior isn't the thin research itself, it's the model's response to being told about it: every one of the 8 thin_coverage retries got the same canned non-response verbatim ("No further tool calls needed... research scope is complete... complete with explicit acknowledgment of gaps") instead of either re-delegating differently or actually writing the honest-partial report the completion-check nudge was asking for. This is the SAME non-convergence pattern already flagged above (the 10x redundant write_workspace_file case) showing up in a third shape: doesn't recognize a real gap needs a different action, just repeats a canned "I'm done" response until the retry budget hard-stops it. Two contributing factors, kept separate from the model verdict since they're infra, not model quality: (1) a real MCP bug independent of the model — brave_web_search's country parameter enum (@brave/brave-search-mcp-server, settings.mcp_servers) does NOT include CO (confirmed via the literal rejection error, tool_error_samples: "Invalid value for 'country' ... 'CO' is not in ['AL..."), so Colombia-targeted searches using an ISO alpha-2 country filter fail outright — a real gap worth a small fix (drop/remap the country param, or catch and retry without it) independent of which model is running; (2) one read_workspace_file/ grep_workspace_file call hit a not-found error on a source filename, the same known fuzzy- filename class already documented elsewhere in this file. Neither infra issue excuses the model's response, though: gpt-oss:20b's own re-runs on this exact query have hit partial fetch failures too and still produced a labeled, honest, written report rather than looping on a fixed refusal string. Bake-off conclusion: gpt-oss:20b remains the only candidate of the seven-plus tried so far (qwen3.6, mistral-nemo, Gemma 4 12B, Bonsai-8B, granite3.1-dense, phi4-mini, qwen3:4b) with a full, real, benchmark-scale pass.

• qwen3:8b — new candidate found and tried 2026-07-18, DISQUALIFIED, same failure class as qwen3:4b. Surfaced by a research pass for tool-calling-capable Ollama models not yet tried (Qwen3's 8B dense sibling, NOT the same model as qwen3.6 (35b-a3b, already rejected) or qwen3:4b — distinct checkpoint, in the Ollama library directly, Apache 2.0, ~5.2GB Q4_K_M). Passed the delegate_tasks tool-call smoke test cleanly: real structured 2-task call, correctly shaped task_name/instructions/agent_id, well-specified instructions comparable in detail to Gemma 4's. Derived tag deepdelve-qwen3-8b created (num_ctx 16384, same pattern). Full sales-forecasting benchmark: Report: NOT WRITTEN after 1037.4s, retry budget exhausted (8/8) on thin_coverage, 5 sources fetched (better than qwen3:4b's 1, still not enough — only 2/6 delegated tasks produced a real source). Same disqualifying shape as qwen3:4b: doesn't act on the completion-check's corrective nudge. Distinctive final-turn behavior worth noting: instead of dispatching a writer role, the model's last response NARRATED full findings.md and final_report.md content inline as chat prose (headers, sections, a "Stop here." sign-off) — neither file exists on disk (confirmed, ls on the run folder). Not the same mechanism as Bonsai-8B's writer-role tool-skip (Bonsai had real FindingsWriter dispatches that skipped the tool call; this never got there, the Planner-level conversation narrated instead of accepting the thin_coverage verdict and letting the engine dispatch a writer for an honest partial artifact). One non-fatal MCP schema mismatch during the run (brave_web_search's result_filter enum rejected an out-of-list value), handled cleanly via the existing detailed-tool-error mechanism, not a contributing cause. qwen3:8b DISQUALIFIED as a gpt-oss:20b replacement — updates the bake-off conclusion above: 8 candidates tried, gpt-oss:20b still the only full pass. Ministral- 8B-Instruct-2410, watt-tool-8B, and Salesforce Llama-xLAM-2-8b-fc-r were also surfaced by the same research pass but not pulled/tested this session (the latter two are narrow function-calling finetunes, a real risk for the writer role per this project's own repeated lesson — not worth GPU time until a general-purpose candidate looks more promising than the two Qwen3 sizes just tried).

• llama3.2:3b — new lightweight candidate tried 2026-07-18, DISQUALIFIED at the tool-call schema stage, root-caused rather than assumed. Surfaced by a research pass specifically for models LIGHTER than the two disqualified Qwen3 sizes, targeting their exact failure mode ("doesn't follow a corrective instruction precisely") rather than a raw-capability gap — Llama 3.2 3B has the best documented IFEval/BFCL combination in its weight class and a native Ollama tool-call template. Derived tag deepdelve-llama32-3b created (num_ctx 16384). Real, structured tool_calls responses (correct function name, valid top-level JSON) — but the tasks array parameter's VALUE is itself a JSON-encoded STRING ({"tasks": "[{\"task_name\": ...}]"}) instead of a real array, reproduced 3/3 times against the exact delegate_tasks schema. Root-caused, not just observed: recreated the identical Pydantic model agent_framework builds from delegate_tasks(tasks: list[dict])'s own type hint and fed it the same malformed value — confirmed real rejection (Input should be a valid list [type=list_type]), the same message DeepDelve's own "detailed tool-call validation errors" feature would show the model live. Then simulated the full round-trip: fed the model that exact real error and asked it to retry. Result: it did not correct the array, it abandoned structured tool-calling entirely — wrote a Python code snippet in chat prose, then a hand-typed single-quoted (invalid JSON) pseudo-call as plain text. Same "narrate instead of call" disqualifying class as granite3.1-dense/phi4-mini, just reached one message later (after a correction) instead of immediately. DISQUALIFIED without a full benchmark run — same evidentiary bar this project already applies to schema-stage rejects (devstral, hermes3, etc. in the README table): a model that gets WORSE after seeing the exact right correction isn't worth 20-40 GPU-minutes to find out how it does on the full pipeline.

  • Documentation check (2026-07-18): this is a known, unresolved upstream Ollama limitation, not something specific to this project's integration or to llama3.2 itself. ollama/ollama#6155 ("Support Nested Parameters for Tools," filed Aug 2024, still open, no maintainer fix) documents the identical stringified-nested-array symptom across llama3.1:8b/70b, mistral-nemo, and llama3-groq-tool-use — an Ollama-side parser/serialization limitation with array/nested-object tool parameters generally, not a single model's chat-template quirk. ollama/ollama#7860 separately documents Llama 3.2 mangling SCALAR parameter types too (ints returned as strings), so this model has broader type-fidelity problems beyond just nested arrays. No documented workaround exists anywhere in the issue tracker or community discussion (checked #6155, #7860, #10552, #11805, #13519) — the only mitigation found anywhere is a LangChain-side client library that re-parses shallow string-encoded JSON arguments after the fact, not a model-side or Ollama-side fix. Because this is a shared Ollama-level limitation (not model-specific), it could in principle affect ANY future candidate with an array-typed delegate_tasks argument, including intermittently against models that otherwise pass — worth remembering as context if a future candidate shows an occasional, not-fully-reproducible schema hiccup.
  • Deferred, not implemented, structural candidate: a defensive "if a list-typed tool argument arrives as a JSON-encoded string, parse it before validation" tolerance would be a generically useful robustness improvement given the above (helps any model that hits this known Ollama-side quirk, not just llama3.2) — but it wouldn't have rescued llama3.2:3b itself (the disqualifying event is the collapse-into-narration on retry, which happens after the string would already have been coerced), and the only interception point found (agent_framework.FunctionTool.invoke's internal Pydantic validation, built from delegate_tasks's own type hint) would require either widening that hint in a way that also changes the JSON schema shown to every OTHER model including the working default, or monkeypatching vendored agent_framework internals — real blast radius against a function every single model/role depends on. Not attempted this session; flagged for a dedicated reviewed session if a future candidate's only blocker turns out to be this exact quirk.
  • Ollama-alternative backend research (2026-07-18), conclusion: don't switch, not yet justified. Since the array-stringification bug looked Ollama-level rather than model-level, researched whether switching the local inference backend entirely would sidestep it. ollama/ollama#6155 is actually CLOSED (merged via PR #13508, Dec 2025) — but the merged fix only adds nested-object SCHEMA DEFINITION support (api/types.go's Properties field, so Ollama can describe a nested schema to the model); it does NOT touch the separate response-parsing path that turns the model's raw tool-call text back into arguments, which is exactly where llama3.2:3b's failure was reproduced live on this project's installed version (0.31.2, months newer than the merge). Worth a fresh, narrower upstream issue with the exact repro if this recurs. llama.cpp's own server is not a cleaner alternative — its issue tracker has its own open, unfixed array/nested-object tool-call serialization bugs (ggml-org/llama.cpp#21384 closed as not-planned, #20198/#22072 open, #20359 on malformed JSON for large payloads) on essentially the same grammar/parsing machinery class, not a structurally different guarantee. Native (non-Docker) vLLM on ROCm is now realistic on this exact card (AMD ships installable ROCm wheels as of Jan 2026, gfx1200 on the officially supported list for ROCm 7.2+) — a real change since the earlier vLLM investigation, which only ever hit the NTFS+Docker-overlayfs blocker and never tried a native pip install. vLLM's grammar-constrained structured-output tool-calling (token-level schema constraint, not post-hoc regex/PEG re-serialization) is theoretically more robust for array-of-objects arguments than either Ollama's or llama.cpp's approach, but no direct comparative evidence was found confirming it actually avoids this exact failure mode — the recommendation is theoretical, not proven. Model-weight storage on the NTFS mount is a non-issue for any backend (the NTFS/symlink constraint was specific to Python venvs and llama.cpp's/HF's own auto-download symlink cache; a directly-specified local GGUF/safetensors file path has no symlink requirement). Conclusion: not worth migrating now — qwen2.5:3b-instruct and both qwen3 sizes already don't hit this bug on Ollama as currently installed, so there's no live blocker actually forcing a backend change; revisit only if a future candidate's sole blocker turns out to be this exact array-stringification bug with no working Ollama-served alternative.
  • HANDS-ON CROSS-BACKEND EXPERIMENT DONE (2026-07-18) — CONCLUSIVE: the bug is MODEL-side, not Ollama-side. The prior research pass above was necessarily theoretical (no direct test of the actual failure). Ran a real, controlled A/B: downloaded llama.cpp's official prebuilt ROCm 7.2 release (ggml-org/llama.cpp tag b10068, llama-b10068-bin-ubuntu-rocm-7.2-x64.tar.gz — matches this card's gfx1200/ROCm 7.2+ support directly, no build needed) and ran llama-server --jinja (the model's own embedded chat template, confirmed genuine by reading the GGUF's tokenizer.chat_template metadata directly — the real official Meta Llama tool-calling template, not a generic fallback) against the SAME two models already tested on Ollama, GGUF weights pulled fresh from Hugging Face onto the NTFS mount (/mnt/nuevovol/llm-models/, confirms model-weight NTFS storage really is a non-issue for any backend as predicted — plain hf_hub_download calls, no symlink involved). Result: llama3.2:3b reproduces the IDENTICAL array-stringification bug 3/3 times on llama.cpp's own server ({"tasks": "[{...}]"}, a JSON-encoded string, not a real array) — same failure, completely different serving software, different parser, different (grammar-constrained, not regex-based) tool-call extraction mechanism. qwen2.5:3b-instruct produces a clean, correctly-typed array 3/3 times on the exact same llama.cpp server — matching its behavior on Ollama. This is a clean, well-controlled result: same backend, same template-authenticity check, one model fails consistently and the other passes consistently — the variable that predicts the bug is the MODEL, not the serving software. Directly answers the concern that this project might be missing out on real model options because of an Ollama-specific defect: it isn't one. A model that fails this way on Ollama will very likely fail the same way on llama.cpp or (by extension, though not directly tested) vLLM, since the failure tracks the model's own learned generation behavior around nested-array arguments, not a serving-layer parsing quirk. Conclusion reinforced, now with direct evidence instead of just literature research: no backend migration would have saved llama3.2:3b, and there's still no live blocker forcing one for any candidate that currently works. llama.cpp binary and both test GGUFs left on the NTFS mount (/mnt/nuevovol/llm-models/, ~4GB total) in case a similar quick cross-backend check is useful again later — trivial against the drive's 1.1TB free.
  • Third backend added to the A/B, same day: native (non-Docker) vLLM-on-ROCm, not just theorized — actually run. The "no direct evidence" caveat above was addressed head-on rather than left as a gap. Built a real native vLLM install (vllm==0.25.1+rocm723, the official AMD-published ROCm wheel from wheels.vllm.ai, matching this exact gfx1200 card) in a throwaway venv on the root disk (per the existing venv-must-be-on-ext4 rule; NTFS still can't hold the Python venv's symlinks). Getting it running required manually resolving a long chain of missing shared libraries one ldd sweep at a time (no root/sudo available in this environment) — OpenMPI runtime libs, several ROCm math libs (rocFFT, rocRAND, rocSPARSE, hipFFT/hipRAND/hipSPARSE/hipSOLVER/hipSPARSELt, RCCL, rocm-core, roctracer/libroctx) not present anywhere on this system outside Ollama's own bundled, incomplete copy — each fetched directly as a .deb from repo.radeon.com's public ROCm 7.2 apt pool and extracted with dpkg-deb -x into a scratch dir (no apt install/root needed), then wired in via LD_LIBRARY_PATH/ROCM_HOME. Confirmed working: torch.cuda.is_available() True, gcnArchName correctly gfx1200. Served unsloth/Llama-3.2-3B-Instruct (an ungated mirror; the official meta-llama repo is gated and wasn't authenticated in this environment) via vllm serve --enable-auto-tool-choice --tool-call-parser llama3_json — vLLM's own purpose-built parser for the Llama 3.x tool-call format, its most favorable possible configuration for this exact model family. Result: identical bug, 3/3 — {"tasks": "[{...}]"}, a JSON-encoded string, not a real array, exactly matching Ollama and llama.cpp. Three independent backends, three structurally different tool-call extraction mechanisms (Ollama's Go templating, llama.cpp's GBNF grammar, vLLM's own structured-output constraint engine with a model-family-specific parser) — same model, same failure, every time. This is now definitive, not theoretical: the bug is 100% attributable to llama3.2:3b itself, and no realistic backend migration would recover it. Root-disk cleanup done immediately after (venv + manually-fetched ROCm libs removed, ~16GB freed, root back to 61GB free) — same disk-hygiene lesson as the GRPO smoke test session, a throwaway experiment venv doesn't linger.

• qwen2.5:3b-instruct — new lightweight candidate tried 2026-07-18, DISQUALIFIED, different failure class than llama3.2:3b. Second candidate from the same "lighter than the disqualified Qwen3 sizes" research pass. Passed the delegate_tasks schema test cleanly, 7/8 across two batches (one silent empty-response outlier, otherwise a real, correctly-typed array every time) — does NOT reproduce the array-stringification bug that disqualified llama3.2:3b, so worth the full benchmark run this time. Derived tag deepdelve-qwen25-3b (num_ctx 16384). Full sales-forecasting benchmark: Report: NOT WRITTEN after 254.6s, retry budget exhausted (8/8) on missing_findings — much faster to fail than either Qwen3 size (255s vs. 1000+s), because the failure surface is different and narrower: real research DID happen (2 sources fetched cleanly, en.wikipedia.org/wiki/Heuristic_(computer_science) and .../Public_holidays_in_Colombia, 0 search failures), and the Planner correctly stopped delegating and let the engine dispatch FindingsWriter — but FindingsWriter never successfully produced findings.md across all 8 attempts. Confirmed via _run_state.json: 20 tool errors recorded, the overwhelming majority "'findings.md' not found" from ReviewFix_attempt{1..8} trying to read a file that was never written. This is the same root cause already documented for Bonsai-8B (writer-tier sub-agent "Finishes" its turn without ever successfully calling write_workspace_file) — and it's a second live confirmation that the 2026-07-14 hardening (_dispatch_writer_review_fix's read-quota-delta cross-check, commit bfd2cd5) is working exactly as designed: every ReviewFix attempt got a genuine, correctly- surfaced "file not found" error rather than a false "REVIEW: CLEAN" on a file that was never read. qwen2.5:3b-instruct DISQUALIFIED as a gpt-oss:20b replacement — updates the bake-off conclusion: 10 candidates tried total (counting llama3.2:3b), gpt-oss:20b still the only full pass. Net signal from both new lightweight candidates this session: smaller models in the 2-5B range are failing at TWO distinct, well-characterized points in the pipeline (schema-stage double-encoding for llama3.2:3b; writer-role tool-call omission for qwen2.5:3b-instruct and Bonsai-8B) rather than one common weakness — there's no single fix that would rescue this whole size class, which is itself useful evidence for the fine-tuning plan above: qwen3:4b remains the better fine-tuning target precisely because ITS failure (thin_coverage non-convergence) is the single narrowest, most well-characterized gap of any candidate tried so far.

• gpt-oss:20b re-confirmed live (2026-07-14), same benchmark, same session: deepdelve-gpt-oss produced a real, grounded final_report.md in 1079.1s, 15 sources fetched, 0 search failures, passing the NLI entailment check along the way (one nli_unsupported retry, corrected). First fresh confirmation this session that the documented default actually still passes end-to-end, directly alongside the same-day Bonsai-8B/qwen3:4b/Gemma-4 attempts on the identical query — the only one of the four to produce a written report at all. Content covers the heuristic- optimization side of the query well (PSO, GA, moving-average, rule-of-thumb) but drops the Colombia-specific cultural-context research the run itself actually did earlier (holidays/ paydays from Banco de la República were researched but never made it into the final report) and doesn't surface the gold reference's DL-architecture families — a real report, correctly grounded, but likely a partial (not top) score against the full manual rubric if formally scored. Not manually scored this session (would need a careful pass against eval/reference/sales_forecasting_deepseek.md).

  • Formally scored 2026-07-18, per eval/sales_forecasting_benchmark.md's rubric: 6/10 ("usable with manual verification"). Tier 1 structural integrity 2/2 (findings.md+final_report.md both exist, 18/18 fetched URLs clean, none flagged stub among the ones actually cited, run converged clean by completion-check attempt 3). Tier 2 architecture coverage vs. reference 0/2: the report covers 3 optimization/feature-search heuristics (GA time-lag selection, TS_Adam, Randomized Uphill Climbing) but none of the reference's 4 forecasting-architecture families (TFT, N-HiTS, DQN, EventCast/multimodal) — a real, grounded, but structurally different literature set, not a fabrication. Tier 3 heuristic-optimization coverage 2/2 (GA applied to LSTM hyperparameter tuning, matching the query's actual framing). Tier 4 Colombia cultural context 0/2, and this is the more interesting result: _run_state.json shows timeanddate.com/holidays/colombia/2024 WAS fetched cleanly (not a stub) alongside one stubbed ADP payroll-calendar fetch, yet neither findings.md nor final_report.md mentions Colombia even once — a second, independent live confirmation of the shared-quota-pool starvation bug logged above (a different run, different model context than the original find), not a new bug. Tier 5 quantitative grounding 2/2 (every reported figure traces to an *Evidence:* line from a real fetched source). Net: the defense layers correctly prevented fabrication on a topically disjoint literature set; the score ceiling here is entirely the quota-starvation bug, not a grounding failure.

• Heterogeneous role tiering (option 2 above) — implementation and A/B test detail. A UNIFORM small-model dispatcher was tried and rejected 2026-07-11 (nemo scored 2/10 across every role); this instead tiers by role, keeping gpt-oss:20b for Planner/Builder/FindingsWriter/PeerReviewer (the roles needing multi-step self-correction) and routing WebSearcher/AcademicSearcher/ DocumentAnalyzer/DataAnalyzer to a new optional settings.specialist_model.

  • Implementation: _build_client(model_override=None) (src/engine/orchestrator.py) now takes an optional model override; create_local_agent builds a second client only when specialist_model is set and differs from api.openai_model (a no-op object-reuse otherwise); _run_single_task picks the specialist client when agent_id is in the new _SPECIALIST_MODEL_ROLES set (the deliberate complement of the existing _NON_RESEARCH_DISPATCH_ROLES). TUI/CLI status lines updated in parity to show <model> (+specialist: <model>) when configured. config_template.yaml documents the key. No SubAgentConfig/Pydantic changes needed — the routing decision lives entirely at the one dispatch point. test_structural_checks.py passes unchanged.
  • Design flaw, foreseeable before any A/B test ran — added retrospectively 2026-07-21, per user pushback that this should have been caught at design time, not after measuring it. The 2026-07-18 "agreed order" (structural fix → tiering → fine-tuning → stay on gpt-oss) approved trying tiering as a strategy step, not this specific pairing's reasoning — that reasoning was never spelled out before implementation. The VRAM probe below was run BEFORE writing code and already showed the disqualifying fact: this card cannot hold gpt-oss:20b and any second model resident at once. Given that, pairing a "heavy" coordinator model that must stay loaded for Planner/Builder/FindingsWriter/PeerReviewer with a "light" specialist for the remaining roles was never actually lighter in aggregate VRAM terms — gpt-oss:20b doesn't get unloaded between specialist calls, so the specialist tier only adds a second model competing for the same fixed budget, guaranteeing constant eviction/reload thrashing regardless of which small model was chosen. The 4.2x slowdown below is the confirming measurement of a result the probe's own numbers already implied — it should have been treated as a go/no-go gate before running the A/B, not just a footnote alongside it. Standing implication for any future specialist-model retry: before implementing, check whether the specialist's footprint fits ALONGSIDE the coordinator model's resident footprint (not just its own footprint against the total VRAM budget) — if the coordinator model can't be unloaded between specialist dispatches, tiering cannot reduce peak VRAM pressure, only add to it.
  • VRAM probe done BEFORE writing any code: confirmed live via ollama ps/rocm-smi that this card does NOT keep two different Ollama models resident simultaneously — gpt-oss:20b (12GB) and qwen3:4b (5.1GB loaded, inflated by KV cache) together exceed the ~15.9GiB budget, so Ollama evicts the previous model on every switch. Measured reload cost: ~5-23s per switch.
  • Real timed A/B run (2026-07-18), same sales-forecasting benchmark, gpt-oss:20b + specialist_model: deepdelve-qwen3-4b, confirmed via ollama ps mid-run that both roles really did route to their intended model. Result: 4513.1s (75.2 min) vs. the pure gpt-oss:20b baseline's 1079.1s — 4.2x SLOWER, not faster, driven by the reload tax compounding across an unusually retry-heavy run (thin_coverage → missing_findings → missing_artifact before converging) plus qwen3:4b needing repeated redispatches (background_heuristics#2/#3/#4) to produce anything usable for its assigned angle. Worse, the run converged CLEANLY (no fabrication, real grounding, Report: written) but the content itself silently dropped the query's entire main topic: findings.md and final_report.md are 100% about Colombian holidays/payroll, with ZERO mention of heuristic algorithms or deep learning, despite _run_state.json confirming the specialist model DID eventually fetch two genuinely relevant real sources for that angle (sciencedirect.com/.../S1546221825008872, forecastio.ai/blog/time-series-forecasting) that even show up in RunState.data["findings"]. The content existed; the writer-tier synthesis (on gpt-oss:20b, the coordinator model, not the specialist) dropped it anyway. This is a NEW instance of the pattern already tracked elsewhere in this file (real fetched content silently absent from final synthesis) — previously always tied to an observable quota-exhaustion trigger, but no quota exhaustion is visible in this run's own attempt log, suggesting the underlying issue may be a broader writer-tier prioritization/attention problem, not solely the already-scoped quota-fairness bug. Not investigated further this session — flagged as a new, distinct candidate worth its own root-cause pass.
  • Conclusion: tiering the code is correct and reusable, but THIS pairing (gpt-oss:20b+qwen3:4b) on THIS hardware is a net loss — slower AND lower quality than just running gpt-oss:20b alone. specialist_model left unset in the live config (defaulting to today's single-model behavior). Worth retrying only if: a specialist model with a smaller combined VRAM footprint (fits alongside gpt-oss:20b without eviction) is found, or the newly-surfaced writer-tier content-dropping bug gets root-caused and fixed first — as scoped, tiering does not currently deliver the hoped-for benefit.

vLLM bake-off re-test — candidate verdicts, appended as concluded (moved from Pending during the 2026-07-21 ROADMAP status audit; the pre-flight/plan framing sentences are kept as-written since they're accurate history, even though the umbrella task itself is not finished — see Pending for what's still open):

• Re-run the full 11-candidate local-model bake-off via vLLM instead of Ollama — planned 2026-07-21, not started. Two independent, confirmed Ollama-serving-layer bugs (the think-mode passthrough failure documented in this file's Qwen3-family entry above, and the pre-existing ollama/ollama#6155 nested-array tool-parameter stringification bug affecting mistral-nemo, llama3-groq-tool-use, and llama3.2:3b) mean several of README.md's 11 bake-off disqualifications may reflect Ollama's own serving bugs rather than genuine model incapability. Full plan (per-candidate VRAM/quantization feasibility, tool-parser mapping, execution order, and the real blockers found during research — this vLLM install has no GGUF support at all, bitsandbytes isn't installed, Bonsai-8B's quant type is unrecognized by vLLM, the GRPO fine-tune's merge checkpoint is gone from disk, qwen3.6/Gemma 4 12B's HF availability is unconfirmed) written to ~/.claude/plans/moonlit-plotting-simon.md. Scoped explicitly as a multi-session effort, not a single sitting.

  • Both pre-flight checks DONE, 2026-07-21 — cleared, execution can proceed.

    • HF repo IDs confirmed to exist: google/gemma-4-12B-it (official Google org, not the community SetneufPT GGUF reupload originally used) and Qwen/Qwen3.6-35B-A3B — plus a bonus find, a pre-quantized Qwen/Qwen3.6-35B-A3B-FP8 checkpoint exists too, which helps the MoE-fits-at-all question the plan flagged as unconfirmed.
    • bitsandbytes spiked on ROCm, real functional pass, not just import success. Installed cleanly (pip install bitsandbytes — wasn't present before). Checked bitsandbytes' own support matrix first (not assumed): this GPU's gfx1200 target (confirmed via rocminfo, RX 9060 XT) IS on their officially-supported RDNA list. Ran a real discriminating test on mistralai/Mistral-7B-Instruct-v0.3 (--quantization bitsandbytes --load-format bitsandbytes): failed to fit in a deliberately tight 0.3 gpu_memory_utilization budget (~5.1GB) with "no available memory for cache blocks," then succeeded cleanly at 0.45 (~7.7GB total, weights+KV). Since the real bf16 checkpoint is 13.5GB, fitting inside 7.7GB total is only possible if the weights are genuinely quantized to roughly 4-5GB, not silently loaded at full precision — confirmed real, working 4-bit quantization on this hardware, not a silent no-op. Correct generation output ("Paris") and a real structured tool_calls response (via mistral_tool_parser.py) both verified. The 8-candidate quantization bucket in the plan above is now trustworthy to execute.
    • New operational lesson, found mid-spike, applies to EVERY vLLM launch/kill in this project from now on: killing an already-running (not self-crashed) vllm serve process with pkill -9/kill -9 reliably orphans its VLLM::EngineCore child (confirmed via ps -ef --forest: EngineCore is a real child of the APIServer process, spawned via Python multiprocessing with no death-signal hookup to its parent) — SIGKILL can't be trapped, so vLLM's own shutdown code never runs to tear down the child, and it keeps holding VRAM indefinitely. This explains every "stale EngineCore still holding Xgb" gotcha hit repeatedly this session (MiniCPM5-1B twice, MiniCPM3-4B, this spike's first kill attempt). Fix: use plain SIGTERM first (pkill -f "vllm serve ...", no -9) and give it a few seconds — this lets vLLM's own cleanup path run, confirmed via a clean SIGTERM kill on this exact spike's running server leaving zero orphan afterward. Only escalate to -9 on whatever's left if rocm-smi --showpids still shows something after a graceful SIGTERM attempt.

  • Next session/execution starting point: proceed straight to the plan's per-candidate procedure, in its documented priority order (mistral-nemo:12b and llama3-groq-tool-use:8b first — highest information value, directly implicated in the confirmed #6155 Ollama bug).

  • mistral-nemo:12b re-test, 2026-07-21 — BLOCKED, a real infrastructure incompatibility, not a capability verdict. mistralai/Mistral-Nemo-Instruct-2407 (not gated, native MistralForCausalLM support, confirmed real bf16 checkpoint) loaded cleanly via ~/.venvs/vllm with bitsandbytes 4-bit quantization (same proven path as the pre-flight spike) and the mistral tool-call parser. Isolated tool-call smoke test with DeepDelve's real nested delegate_tasks schema PASSED cleanly: a genuine structured array (tasks: [{...}]), not the stringified-JSON shape from Ollama's #6155 bug — direct confirmation the bug is absent on this backend, exactly as expected. But the full DeepDelve run failed immediately on its very first request, 100% reproducibly, with 400: "chat_template is not supported for Mistral tokenizers." Root cause traced to vLLM's own source (vllm/tokenizers/mistral.py::validate_request_params): vLLM's Mistral-native tokenizer class unconditionally REJECTS any request containing chat_template_kwargs — and DeepDelve's _get_default_options() (src/engine/ orchestrator.py) unconditionally SENDS chat_template_kwargs: {"enable_thinking": ...} on every single dispatch, regardless of model family. This is a hard, structural mismatch between DeepDelve's client and any genuine Mistral-family repo served via vLLM's native tokenizer mode — not something a benchmark run can work around.

    • Both alternate tokenizer modes tried and both failed for a different reason each: --tokenizer-mode auto still auto-detects Mistral's native tokenizer class from the repo's shipped tekken.json/params.json (same rejection, unchanged). --tokenizer-mode hf fails at engine startup entirely with AttributeError: CachedMistralCommonBackend has no attribute is_fast — this repo's shipped tokenizer files aren't compatible enough with vLLM's HF-tokenizer-mode wrapper either. No third option exists in this vLLM version.
    • Verdict: BLOCKED, not disqualified and not re-testable as-is. MiniCPM3-4B's "genuinely open infrastructure question" framing applies here too — this never reached testing mistral-nemo's actual research/delegation behavior at all, so the original Ollama-served 2/10 score stands unconfirmed/unrefuted by this attempt. Real fix, if this is worth pursuing later, is a DeepDelve-side change: make _get_default_options()'s chat_template_kwargs injection conditional (e.g., skip it for models that don't need/support the enable_thinking toggle at all, or catch/strip on this specific 400 and retry once) — out of scope to hack into production code mid-benchmark without the user's sign-off, since it touches every model's request path, not just this one candidate's. Confirmed to affect every other genuine Mistral-family repo in this project's candidate list, not just a possibility: checked mistralai/Devstral-Small-2507 directly — same MistralForCausalLM architecture, same shipped tekken.json (Mistral's native tokenizer format), so devstral:24b's re-test would hit the identical 400 block. mistral:7b-instruct-v0.3 (already spiked earlier this session for the bitsandbytes pre-flight check, same mistralai org/format) would too. All three Mistral-family candidates in the vLLM re-test plan are blocked by this same issue — none re-testable until DeepDelve's client-side fix above lands. Cleanup: config reverted to deepdelve-gpt-oss:latest/rag_cache: enabled: true, vLLM server shut down cleanly (SIGTERM, confirmed zero orphan both times it was killed during this attempt). UPDATE, same day, later: the client-side fix landed (settings.skip_chat_template_kwargs, commit 9c8111f — see the dedicated "even later" History entry above). mistral:7b-instruct was re-tested with the fix applied and got past this exact block, confirming the fix works — but mistral-nemo:12b itself has NOT yet been re-run with the fix applied. Its verdict here remains BLOCKED-not-yet-re-tested, not assumed to inherit mistral:7b-instruct's own not_delegated result — that's a separate, still-open re-test opportunity (devstral:24b likewise unblocked and untested).

  • llama3-groq-tool-use:8b re-test, 2026-07-21 — DISQUALIFIED on real, docs-grounded evidence, NOT a serving-stack artifact. Groq/Llama-3-Groq-8B-Tool-Use (not gated, native LlamaForCausalLM, real bf16 checkpoint). Its own native max_position_embeddings is only 8192 — below the project's ~16K floor, but this is a permanent model-level training fact, not a hardware-forced squeeze, so the "discard outright below 16K" standard point 6 does NOT apply here (clarified in that point above) — tested at its real native 8192 ceiling instead.

    • First smoke test (plain OpenAI-style tools= + tool_choice: "auto") failed outright: the model narrated a plain-text answer, never attempting a tool call at all. Root cause checked directly, not assumed: this repo's own tokenizer_config.json chat template has ZERO tool-rendering logic ('tools' in chat_template is False) — a bare vanilla Llama-3 template. vLLM's tools= parameter never got rendered into the prompt in any form this model could act on, so this first result wasn't a real capability test yet.
    • Read the model's own HF README (credits NousResearch for this exact tag convention) and manually built its documented raw system-prompt format (<tools>...</tools> + <tool_call>...</tool_call> instructions embedded directly in the system message, bypassing the broken auto-render path). Result: 3/3 samples (including the model card's own recommended temperature=0.5, top_p=0.65) produced genuinely well-formed, correctly-structured JSON with a real nested tasks array (#6155-class bug confirmed absent) — but the model consistently omitted the required <tool_call>/</tool_call> XML wrapper tags every single time.
    • Caught mid-investigation, per the user's explicit correction: tried priming the assistant turn with a literal <tool_call> opening tag as a fix — an UNSOURCED generic technique, not verified against this model's own documentation first. User stopped this and asked directly whether the model's docs had actually been consulted; they hadn't. Went back to primary sources instead: checked Groq's own cookbook (documents their HOSTED API, a different serving stack, not applicable to local vLLM hosting), then found and read NousResearch's own Hermes-Function-Calling reference repo (the exact upstream implementation this model's tag convention is credited to) and its real parsing code — utils.py::validate_and_extract_tool_calls requires the literal <tool_call> XML element via root.findall(".//tool_call") and returns zero tool calls without it. Confirmed vLLM's own bundled hermes_tool_parser.py requires the identical <tool_call> token (same tool_call_start_token check before extraction) — so this isn't a vLLM-specific integration gap either; both the credited reference implementation and vLLM's own parser agree the tags are mandatory.
    • Verdict, now grounded in real evidence rather than assumption: the model's underlying JSON-generation quality is genuinely good (correct structure, real BFCL-consistent capability, no #6155-class bug) — but it does not reliably emit the <tool_call> wrapper tags any correctly-built Hermes-style parser requires to extract a real structured tool call, confirmed against 2 independent authoritative sources (the credited upstream reference parser and vLLM's own bundled parser), not just this session's own serving setup. This is a genuine, dual-confirmed disqualification, not the Ollama #6155 artifact this candidate was originally suspected of — the original schema-stage rejection stands, now on firmer evidence than before. Cleanup: server shut down cleanly (SIGTERM, zero orphan), no config change needed (never got far enough to wire DeepDelve's config at all — disqualified at the isolated smoke-test stage, per the plan's own step 3 evidentiary bar, no full benchmark run spent).

• qwen3:8b vLLM re-test, 2026-07-21 — KILLED mid-run, real DeepDelve-side fabrication bug found and fixed, no verdict on the model yet. Loaded via ~/.venvs/vllm, nothink mode confirmed clean via direct curl before running (README's qwen3-family think-mode bug is Ollama's own serving-layer defect, not the model's — already confirmed absent on vLLM the same session). Run was genuinely progressing (3rd delegation round, 18 fetched URLs, 19 findings, clearly better-behaved than any MiniCPM candidate) when a user-requested cross-check against the real sources/ folder caught a real integrity problem: only 15 files on disk vs. 18 claimed fetched_urls and 19 findings, and 5 of 19 findings had a fabricated source_url — a leaked task/instruction name string instead of a real URL. Run killed before reaching FindingsWriter; no verdict reached on qwen3:8b itself.

  • Root-caused, 2026-07-21, confirmed model-agnostic: _run_single_task's add_finding fallback (src/engine/orchestrator.py) used the bare task_name as source_url whenever a dispatched task (any Analyzer-tier call, by design) fetched no URL of its own, with no marker distinguishing it from a real citation. _build_findings_source_material (src/engine/completion.py) then rendered every finding identically as ### Source: {source_url} regardless of whether that value was a real URL or the placeholder — FindingsWriter (any model, on any backend) had no structural signal to tell them apart. This is the same mechanism regardless of which model is serving FindingsWriter, so it was not qwen3:8b-specific and would have equally exposed every other vLLM re-test candidate still to come.
  • Fixed, commit 0852cc4: (1) orchestrator.py now recovers the real reference URL a Searcher handed its Analyzer (already extracted for the reconstructed-URL check, now computed unconditionally rather than gated behind grounding_check.enabled) before ever falling back to task_name; (2) _build_findings_source_material never renders a non-http(s) source_url as a ### Source: ... entry anymore — such findings are named in a separate, explicitly non-citable list instead, with instructions not to invent a source for them. Matters more given this project tiers some writer roles onto smaller specialist models (settings.specialist_model), which are less likely to infer the ambiguity on their own. test_structural_checks.py extended (_findings_uncited_fallback_scenario) and existing filename-scenario assertion corrected to match the new behavior; both pass.
  • No past verdict in this file was corrupted by this bug: MiniCPM5-1B's disqualification was zero delegate_tasks calls (never reached findings.md), llama3-groq-tool-use:8b's was a missing <tool_call> wrapper (never reached research), mistral-nemo:12b's was a first-request 400 (never reached research) — none of the currently-closed vLLM re-test verdicts relied on findings.md content, so none need re-opening.
  • Next step: retest qwen3:8b fresh now that the bug is fixed — this candidate is the most informative next run precisely because it's the one that surfaced the bug.

• qwen3:8b vLLM retest, 2026-07-24 — DISQUALIFIED, run twice, identical failure signature both times. The fabrication bug that killed the 2026-07-21 attempt was already fixed (0852cc4); this session re-downloaded Qwen/Qwen3-8B (bf16, ~16GB) and served it via ~/.venvs/vllm (vLLM 0.25.1+rocm723) with --quantization fp8 (dynamic on-the-fly weight quantization — the bf16 checkpoint alone needs ~14.9GB against 15.9GB total VRAM, no room for KV cache; fp8 halves weight memory to ~8.8GB, confirmed working via the ROCm-native ROCmFP8ScaledMMLinearKernel, leaving ~4GB for a 16,384-token KV cache).

  • Model Evaluation Standard point 1 confirmed via direct curl BEFORE benchmarking, not assumed: enable_thinking: false (passed as a TOP-LEVEL chat_template_kwargs field over raw HTTP, not nested under an extra_body wrapper — that's an OpenAI Python SDK convention that doesn't apply to a raw request and silently gets ignored, a real mistake caught and corrected mid-session) gives a clean content: "56" with zero <think> leakage and reasoning: null. A separate direct tool-call curl test confirmed clean, correctly-shaped tool_calls output with the hermes parser. Both checks pass.
  • Full single-model benchmark run TWICE against the exact same two-facet stress-test query used for this session's gpt-oss confirmation ("top 5 heuristic algorithms for retail sales forecasting" + "Colombia holiday consumer spending culture"). Both runs: final_report.md was never written, Retry budget exhausted (thin_coverage). Both runs hit the IDENTICAL completion-check attempt sequence (thin_coverage ×2 → untracked_delegation → thin_coverage ×4) — a remarkably consistent, reproducible failure signature, not a fluke. Root behavior in both: once thin_coverage fires, the Planner repeats the SAME narrated summary text verbatim across multiple consecutive completion-check attempts instead of genuinely delegating more research or stopping to let a writer role take over — the exact same non-convergence pattern already documented for qwen3:4b and this candidate's own original (Ollama, think-polluted) verdict, now confirmed on a clean backend with verified nothink mode and correct tool-calling, closing off the "maybe it was Ollama's serving bug" possibility this candidate previously had.
  • Separate, real finding along the way: in both runs, the model repeatedly called the MCP-provided brave_web_search tool with an invalid goggles parameter (a real, optional Brave Search re-ranking feature — the model invented a URL-shaped value for it, e.g. ["https://www.colombia.com"], which the Brave API rejects with a 422), hitting this project's "maximum consecutive function call errors" cap 3+ times per run. Not a hallucinated tool name (as first suspected) — brave_web_search is a real, correctly-configured MCP tool (@brave/brave-search-mcp-server, scoped to WebSearcher); the model just doesn't reliably infer "leave this optional parameter unset" the way gpt-oss does. Fixed same session: WEB_SEARCHER_INSTRUCTIONS (src/prompts.py) now explicitly states goggles is optional and warns against inventing a value for it — a low-risk prompt-level mitigation that should help any model using this tool, not just this candidate. Not yet re-tested against qwen3:8b specifically (the candidate is disqualified regardless, so no further live-testing planned for it) or any other model.
  • Verdict meets the Model Evaluation Standard's point 4 (a discard claim needs more than one run): two clean vLLM runs here, corroborated by two further independent occurrences of the identical non-convergence signature under different conditions (the original Ollama-era qwen3:8b verdict, and qwen3:4b separately) — four total occurrences of the same failure class across different backends/models.
  • Cleanup: vLLM server shut down cleanly (SIGTERM, confirmed zero orphan EngineCore process, VRAM returned to ~1.3GB baseline). Checkpoint deleted (~16GB freed, LLvm Models/hub/models--Qwen--Qwen3-8B). ~/.deepdelve/config.yaml restored to http://localhost:11434/v1 / deepdelve-gpt-oss:latest.

• InternScience/Agents-A1-4B, 2026-07-24 — DISQUALIFIED, run twice (one --depth quick, one --depth standard), same citation-fabrication signature both times. A ~4-5B Qwen3_5- based multimodal/agentic model, tested via Ollama after vLLM was abandoned for it (see below).

  • vLLM abandoned first — a reproducible, fixed-size backend bug, not a config problem. Three launch attempts (--max-model-len 65536, 32768, and 16384 — the last also with --enforce-eager and --max-num-batched-tokens 2048 to rule out CUDA-graph capture and chunked-prefill batch size) all failed identically at engine-init with torch.OutOfMemoryError: ... Tried to allocate 256.00 GiB. The exact same 256.00 GiB figure at three different context/batch configs rules out KV-cache sizing (which would scale with --max-model-len) as the cause — most likely a fixed-size dummy input in this model's multimodal vision-encoder profiling pass hitting a ROCm-specific bug in vLLM 0.25.1. Same failure class as gpt-oss-on-vLLM's earlier permanent abandonment (see the 2026-07-23 section above): a serving-backend/architecture incompatibility, not something fixable by flag-tuning. Not investigated further (would need a vLLM issue search or newer version) — deprioritized per explicit user instruction to fall back to Ollama if vLLM kept causing problems. Cleanup: SIGTERM each time, confirmed zero orphan processes, VRAM back to baseline; checkpoint deleted (8.5GB).
  • Switched to Ollama — official GGUF published directly by InternScience (not a third-party quant): InternScience/Agents-A1-4B-Q8_0-GGUF (~5.2GB, includes an mmproj vision file), pulled via ollama pull hf.co/InternScience/Agents-A1-4B-Q8_0-GGUF:Q8_0.
  • Model Evaluation Standard point 1, confirmed via direct curl against the real /v1/chat/completions endpoint (not just Ollama's native /api/chat): plain completion clean (content: "56"); tool-calling clean (correctly-shaped tool_calls, empty content). Nothink mode NOT honored — neither chat_template_kwargs.enable_thinking: false nor Ollama's own think: false suppressed reasoning; the model always emits a full step-by-step reasoning chain (~200-500 tokens) regardless of the request. Unlike the old Qwen3 think-mode-passthrough bug, this did NOT pollute .content — reasoning stayed correctly isolated in its own .reasoning field the whole time, so this is a latency/token-cost caveat, not a correctness break. Caveat this verdict with a † marker for that reason.
  • Context maximized per explicit user instruction ("set it up with the max context we can"): tested 131072 (fits, 11.6/15.9GB), then the model's native max 262144 (fits but razor-thin, 16.0/15.92GiB, under 1GB free — too risky for real generation-time activation spikes), settled on 200000 (13.95/15.9GB, ~3.1GB free headroom). Derived tag deepdelve-agents-a1-4b created (PARAMETER num_ctx 200000, same pattern as every other candidate).
  • Full benchmark run TWICE against the quantum-entanglement baseline query ("What is quantum entanglement and how is it used in quantum computing?", the same straightforward query used to confirm gpt-oss's clean baseline earlier this session). Run 1 (--depth quick) hit --depth quick's tight max_completion_check_attempts: 2 cap before a fair shot was given — confounded, not a clean verdict on its own, but its single completion-check attempt already showed the fabrication pattern below (unverified_entry_sources:https://en.wikipedia.org/wiki/Bell). Run 2 (--depth standard, full 8-attempt budget) is the clean, decisive result: what_is_quantum_entanglement_and_how_is_it_used_in_20260724_230427, 2656s, 13 sources fetched, 0/12 web-search failures (research itself works fine, tool-calling is reliable in practice, not just in the isolated smoke test).
  • Root cause: FindingsWriter repeatedly fabricates citations to real-sounding but unfetched Wikipedia URLs from its own training knowledge, and PeerReviewer approves the rewrite anyway. Completion-check attempts 3 and 4 both failed with the IDENTICAL unverified_entry_sources problem, citing the exact same URLs (en.wikipedia.org/wiki/Bell ×2, en.wikipedia.org/wiki/Shor) both times — the model regenerates the same fabricated citation across independent rewrite dispatches. ReviewFix_attempt3 and _attempt4 both logged "PeerReviewer found no issues" on the SAME ungrounded content — PeerReviewer isn't catching this specific failure mode; only the separate grounding check (real_grounding_problem's unverified_entry_sources) does. The run correctly terminated via this project's own consecutive-same-problem escalation (2 identical failures in a row → stop wasting budget) rather than burning through all 8 attempts blindly — working as designed, not a bug.
  • Meets the Model Evaluation Standard's point 4 (a discard claim needs more than one run): two independent runs (quick-depth and standard-depth), both citing the exact same fabricated Bell Wikipedia URL — not a single-run fluke.
  • Distinct failure mode from the other disqualified candidates — not the thin_coverage Planner non-convergence pattern seen in qwen3:4b/qwen3:8b; closer to the fine-tuned qwen3:4b GRPO candidate's citation-fabrication failure mode (README's model table), but here it resists correction across multiple write-review-fix cycles even with PeerReviewer's approval, where the fine-tune's version was a partial, improving-but-not-fully-closed gap.
  • ~/.deepdelve/config.yaml left pointed at deepdelve-agents-a1-4b:latest pending the next session's decision on whether to keep testing or revert to the gpt-oss baseline (backup: config.yaml.bak_pre_agentsA1_4b_ollama_20260724).

• MiniCPM5-1B evaluated as both a paired specialist AND a full single-model replacement, 2026-07-20/21 — DISQUALIFIED in both forms, fully closed, see the single-model entry near the end of this bullet for the final, clean, decisive result. User asked to check other MiniCPM4-family options after the MiniCPM4-MCP evaluation below; research (RESEARCH.md's earlier MiniCPM5-1B leaderboard entry) already flagged this as a sub-1.5B model, far below this project's own established capacity floor — but user's explicit framing was "one thing is documentation, another is test, let's try," so tested live rather than ruled out on priors alone.

  • Genuinely simpler integration than MiniCPM4-MCP, confirmed by reading docs first this time (see the correction above about not doing that for MiniCPM4-MCP): MiniCPM5-1B emits XML-style <function name="...">...<param name="...">value</param></function> tool calls (its own chat_template.jinja, read directly), a format close enough to the Hermes/Qwen convention that Ollama's built-in tool-call parser handles it natively — confirmed live, direct /api/chat calls with a tools= param returned correct OpenAI-shaped tool_calls with zero custom proxy code. Plain LlamaForCausalLM architecture (config.json), no custom kernels. OpenBMB has an official Ollama deployment cookbook (docs/deployment/ollama.md) confirming the same integration path and recommended sampling (temperature=0.7, top_p=0.95 no-think mode; 0.9/0.95 think mode) — used exactly as documented, not reverse-engineered.
  • Pulled via ollama pull hf.co/openbmb/MiniCPM5-1B-GGUF:Q8_0 (1.1GB), local tag minicpm5-1b with num_ctx set to 131072 (the model's actual native max per its own config.json's max_position_embeddings, not the cookbook's conservative 8192 example value — same standard applied to MiniCPM4-MCP's 32768 setting earlier).
  • Isolated 5-case smoke test: 5/5 passed, correct function selection, correct abstention on a non-tool question (honestly declined an arithmetic question rather than fabricating an answer — a real, observed instance of a friend's claim that small models given permission to say "I don't know" avoid confident hallucination). One real gap already visible in this isolated test, though: one delegate_tasks call dropped the actual task instructions, keeping only task_name — an argument-completeness weakness, not a format failure.
  • Specialist-role system prompts audited before testing further (per the same read-first correction): WebSearcherInstructions/AcademicSearcherInstructions (src/prompts.py) already explicitly ban finishing a task from "search snippets or your own prior knowledge" — exactly the strategy a friend of the user's independently recommended for small models. No prompt changes were needed; this was already the existing design.
  • Live end-to-end test, same query used throughout this evaluation: the single most favorable MiniCPM result of the day. Correctly found not just the arXiv preprint (2404.02680) but also the actual peer-reviewed PUBLISHED version (ACM DOI 10.1145/3674640) of the same paper, and correctly flagged that a third source (ETH Zürich) self-labels "peer-reviewed" without evidence of external review — a more careful preprint-vs-published- vs-self-claimed distinction than any earlier gpt-oss or MiniCPM4-MCP run made. Rust version (1.97.1, plus beta/nightly) correct. Lowest tool-error count of any MiniCPM variant tested (11, vs. 27-53 for MiniCPM4-MCP's runs), closest yet to the clean gpt-oss baseline (0-8).
  • Real problem, still present: the related_work sub-agent was forcibly aborted TWICE ("Agent trapped in loop. Quota exceeded multiple times for fetch_url_to_workspace") before finally succeeding on retry attempt #4. Same underlying category as MiniCPM4-MCP's issues (not reliably knowing when to stop), different specific shape. The system's own retry/ recovery machinery absorbed this and still produced a good outcome, but first-attempt reliability isn't clean.
  • CORRECTION, 2026-07-21 — every result above was very likely produced in unintended THINK mode, not the nothink mode intended for this role. User asked for an in-depth read of the full official openbmb/minicpm docs/skills tree before treating anything as a settled "discard" — all 23 currently-relevant English docs read directly (main README, all 8 deployment cookbooks, all 5 fine-tuning cookbooks, both minicpm5-deploy/minicpm5-deploy- ollama Agent Skills). Confirmed empirically first: every live /api/chat response from the minicpm5-1b Ollama tag includes a populated "thinking" field with real verbose chain-of-thought, even with a custom Modelfile injecting an empty <think>\n\n</think>\n\n prefix meant to force nothink mode per the model's own chat_template.jinja logic — the injection did not suppress it. This is not a mistake unique to this setup — it's a documented, vendor-acknowledged gap in Ollama's OWN official cookbook and shared by other edge/consumer backends: docs/deployment/ollama.md's own example Modelfile only sets temperature/top_p and comments them "tuned for no-think mode," but never actually injects a <think> prefix into the TEMPLATE block — because "Ollama does not auto-evaluate the GGUF-embedded Jinja chat template; it falls back to the Modelfile's Go TEMPLATE block." Independently confirmed by two OTHER backends' own docs: docs/deployment/mlx.md states plainly "the released chat template auto-injects <think>\n when no system message disables it, so you get think-mode behaviour by default"; docs/deployment/lmstudio.md states LM Studio's chat_template_kwargs.enable_thinking flag is not consistently honored either. Only vLLM and SGLang correctly implement real enable_thinking (both evaluate the actual HF template). Practical tool-calling path found for each: SGLang's MiniCPM5 XML parser only exists on an unreleased main branch (merged 2026-05-22, no pip release yet); vLLM is more practical right now — the repo itself ships the parser file (tool_parsers/minicpm5xml_tool_parser.py, same as the pending upstream PR) loadable into a normal pip install vllm>=0.21 via --tool-parser-plugin, no from-source build needed. Implication: every positive result recorded above (best report quality of any MiniCPM candidate, correct preprint-vs-published distinction) was very likely produced in the heavier, more deliberate think mode, not the fast/latency-bound mode this role actually calls for — so neither the positive results nor the one real weakness (the forced-abort looping) can be trusted as representative of the model's intended operating mode. No discard-or-keep verdict is actually settled; this reopens the question rather than closing it either way. Next step: stand up vLLM with the bridged tool-parser plugin and re-run the same live query in genuine nothink mode before drawing any conclusion. Real hardware caveat checked (not assumed): the user's GPU (RX 9060 XT) is AMD RDNA4, not NVIDIA — vLLM defaults to CUDA-only, but ROCm 7.2 (March 2026) added official RDNA4 vLLM support with "out-of-the-box parity" alongside Ollama/ llama.cpp, so this should work, just via the ROCm-specific install path (Docker image or ROCm wheel) and less battle-tested than the CUDA default every vLLM doc assumes.
  • Status: not yet a final call either way, now for a second, more fundamental reason than "needs more runs" — the model hasn't even been tested in its correct operating mode yet. Best MiniCPM candidate tested by a real margin under think mode; live config left pointed at it (specialist_model: minicpm5-1b) rather than reverted, pending a proper nothink-mode re-test via vLLM before any final call.
  • FINAL VERDICT, 2026-07-20 — genuine nothink-mode retest via vLLM completed; DISCARD for this role. Fixed the pre-existing, broken ~/.venvs/vllm install (missing libopenmpi3t64, then ROCm userspace libs stale relative to the current kernel — fixed via sudo amdgpu-install --usecase=rocm,hip --no-dkms, no kernel module/DKMS involved, fully reversible; a system-level DKMS attempt tried first failed on a genuine kernel-symbol conflict against 7.0.0-28-generic and was cleaned up before this correct approach was found). Launched vllm serve openbmb/MiniCPM5-1B --tool-call-parser minicpm5 --enforce-eager --gpu-memory-utilization 0.20 --max-model-len 16384 (memory-utilization and --enforce-eager both driven down from the plan's defaults after gpt-oss's usual 14.6GB Planner footprint left no VRAM headroom on the 17.1GB card — swapped Planner to deepdelve-mistral-nemo:latest, 7.1GB, for this test only; api.openai_model in ~/.deepdelve/config.yaml is still set to this temporary value and must be reverted to deepdelve-gpt-oss:latest once this entry is read). Confirmed via direct curl with chat_template_kwargs: {"enable_thinking": false} that real nothink mode now works (no <think> leakage) — DeepDelve needed zero new code for this, since orchestrator.py::_get_default_options() already threads enable_thinking through extra_body/chat_template_kwargs at both dispatch sites (line ~678 specialist, ~1274 main). Ran the same live query used throughout this whole evaluation. Result, traced through the raw session log rather than assumed:
    • A genuine content hallucination reached the final report. The model's own first-pass reasoning (session log event 17, well before any remediation pass) already commits to "Blog Rust 1.85.0 - URL: https://blog.rust-lang.org/2025/02/20/Rust-1.85.0/" — a real but stale blog post surfaced by a web-search snippet — as "the latest stable version." It later correctly fetches the actually-current releases.rs page (which plainly states Stable: 1.97.1, confirmed by directly grepping the saved source file), but never revises its earlier claim — instead the final findings/report cite <https://releases.rs/> as the source for the wrong "1.85.0, released on February 20, 2025" value. This is not a misread-ambiguous-source case like the earlier MiniCPM4-MCP filename-hash mistake; the correct number was sitting in a source the model itself fetched and cited, and it reported the wrong one anyway. Traced with certainty to MiniCPM5-1B's own Searcher/Analyzer reasoning, not to the temporarily-swapped mistral-nemo Planner's remediation passes (FindingsWriterFix_attempt2/BuilderFix_attempt3 copied this text forward verbatim from the same flawed findings.md, they did not introduce it).
    • findings.md itself never passed the grounding check on its own terms: it shipped as an "AUTO-RECOVERED DRAFT" (the model narrated the findings as chat text instead of calling write_workspace_file, across the full retry budget) — the salvage path saved the run from an outright missing_findings failure, but the underlying content was never actually verified before being carried into final_report.md.
    • Six identical malformed tool calls: fetch_url_to_workspace called with {"url": "sources/paper_143022.md"} — a workspace-relative path to a file it had already saved, not a real URL — repeated six times with no self-correction, on top of one argument-parsing failure elsewhere. Confusing "fetch a URL" with "read a file I already wrote" is a new, distinct failure shape from anything seen in the earlier think-mode run.
    • Two literal "[Authors' names]" placeholder strings (HAL preprint, ACM paper) also reached the final report uncorrected — a completeness/fabrication-adjacent defect the downstream Builder (mistral-nemo, not gpt-oss, for this run) failed to catch, unlike an earlier same-day run where gpt-oss's Builder did catch and fix an analogous mistake.
    • Required 4 completion-check attempts (not_delegated, missing_findings, missing_artifact, then clean) before the run closed at all. Conclusion: genuine nothink mode is now confirmed reachable and correctly wired end-to-end (infrastructure verdict: works, zero new code needed), but this properly-configured test is, on content reliability, worse than the earlier (unintentional think-mode) run — not better. A single model-generated hallucination that directly contradicts its own cited source, shipped past an already-degraded (auto-recovered, unverified) grounding path, past a Builder that didn't catch it, into the user-facing report, is disqualifying for an unsupervised specialist role regardless of mode. Sub-1.5B parameter budget was flagged as a priors-based concern from the very start of this evaluation (RESEARCH.md); this live result confirms rather than contradicts that prior. Discarding MiniCPM5-1B (both modes now tested) for the specialist role. Cleanup still open: revert api.openai_model to deepdelve-gpt-oss:latest, decide whether to keep or stop the standing ~/.venvs/vllm server, remove specialist_model/specialist_base_url from live config (or point them at a different, larger candidate later).
    • RE-FLAGGED 2026-07-21, per the new "Model Evaluation Standard" section above (point 2, isolation): this verdict does not actually isolate MiniCPM5-1B as the only variable — the Planner/Builder was swapped off gpt-oss:20b onto mistral-nemo:latest to free VRAM for this run (see line ~1186 above), and the uncorrected "[Authors' names]" placeholders were explicitly attributed to that swapped-in Builder failing to catch them, not to MiniCPM5-1B's own output. The traced-to-source Rust-version hallucination and the six malformed fetch_url_to_workspace calls ARE cleanly attributable to MiniCPM5-1B itself (confirmed via the raw session log, not the Builder), so the discard isn't baseless — but it was reached under a confounded pipeline, not a clean one, and should not be read as a fully settled, isolated verdict on the model's own capability. Retest explicitly NOT queued — user decision, 2026-07-21: a clean isolated retest (gpt-oss:20b kept in the Planner/Builder seat) was initially proposed as the outstanding item, but the user rejected pursuing that combination further at all — pairing gpt-oss:20b as coordinator with any small model as a specialist is a strategy the user doesn't want tried again regardless of which small model sits in the specialist slot (see the "Heterogeneous role tiering" closure note above). MiniCPM5-1B's status is therefore left as: discard reached under a confounded test, not fairly re-litigated, and not going to be re-tested in that same paired form. Single-model bake-off run — COMPLETED 2026-07-21, clean and decisive: DISQUALIFIED, no caveats this time. MiniCPM5-1B set as api.openai_model across ALL roles (Planner/ Builder/FindingsWriter/PeerReviewer, not just Searcher/Analyzer) — the same architecture every other bake-off candidate in this section was measured under, and the one evaluation MiniCPM5-1B had never actually had. Ran via ~/.venvs/vllm with the model's real full context (--max-model-len 131072, not the earlier tests' 16384 — the actual max_position_embeddings from the model's own config.json; needed --gpu-memory-utilization 0.9 once nothing else was competing for VRAM, since a stale VLLM::EngineCore process from an earlier launch attempt was still holding 8.1GB and had to be killed first). Confirmed via direct curl before running anything through DeepDelve: nothink mode clean (reasoning: null, zero <think> leakage) — same infrastructure verdict as before, this part was never in question. Ran the exact standing sales-forecasting benchmark prompt (eval/sales_forecasting_ benchmark.md) used throughout this whole bake-off. Result, traced through the raw session log: the model called list_workspace_files once, then think_tool with near-identical reflection text ~20 times in a row, burning its entire think_tool quota (30) without ever once calling delegate_tasks — no Searcher was ever spawned, fetched_urls stayed empty, findings.md was never written. It then asserted "I'll compile the findings and final report now based on the delegated tasks" — a flatly false claim, since nothing had been delegated and no findings existed — repeated verbatim several times in the trailing text output. The engine's own not_delegated completion check caught this correctly (_run_state.json: "No delegate_tasks call was ever made — this looks like an answer from memory, not real research.") and the run terminated with Report: NOT WRITTEN once the overall retry budget was exhausted — no artifact, no fabrication reaching the user, the failure mode this project's completion checks exist to catch, working as designed. This clears every point of the Model Evaluation Standard above with no exceptions: operating mode confirmed via raw API call before scoring (point 1); MiniCPM5-1B was the only variable in the entire pipeline, nothing paired or swapped (point 2, the exact gap the two earlier verdicts had); backend/version stated (vLLM 0.25.1, ROCm, ~/.venvs/vllm) (point 3). A second corroborating run was not initially executed given how early and total the failure was (dead by turn ~20 of a 30-call quota, zero real work of any kind produced). Point 4 corroborated with a real second run, same day: after this session separately found and fixed a real process-hygiene bug (killing an already-running vllm serve with -9 orphans its VLLM::EngineCore child, since SIGKILL can't be trapped — see the "Heterogeneous role tiering"/vLLM re-test entry above), the user asked whether that finding could have contaminated THIS verdict's VRAM/context state. Traced the actual timeline: the one stale-process contamination hit during this evaluation happened BEFORE the scored run (an 8.1GB orphan from a failed 16384-ctx attempt, found and killed before the successful 131072-ctx relaunch that the benchmark actually ran against) — the scored run itself used a clean, correctly-provisioned, freshly-confirmed server throughout, so the original verdict was never actually contaminated. Re-ran anyway as a precaution, from a freshly-clean GPU state (rocm-smi --showpids confirmed zero KFD processes before relaunch), same full 131072 context, same nothink-mode curl confirmation. Result: reproduced the identical core failure — 63 events this time (list_workspace_files x11, think_tool x10, spread across 3 completion-check attempts instead of 1), but again ZERO delegate_tasks calls across the entire run, Report: NOT WRITTEN. Point 4 (discard needs >1 run) is now genuinely satisfied, not just argued around. Final verdict, now doubly corroborated: MiniCPM5-1B is disqualified as a DeepDelve model candidate in BOTH forms tested — paired specialist (confounded, not re-litigated per the user's own decision) and full single-model replacement (clean, decisive, reproduced on an independent run). No further MiniCPM5-1B testing is planned; nothing about this model's evaluation remains open.
  • Cleanup done, 2026-07-21: api.openai_model reverted to deepdelve-gpt-oss:latest, settings.specialist_model/settings.specialist_base_url removed from ~/.deepdelve/ config.yaml (confirmed _build_client's .get(...) fallback in orchestrator.py handles their absence, single-model config resumes cleanly), test vllm serve process killed. ~/.venvs/vllm itself kept on disk — a verified-working general ROCm+vLLM install for this exact GPU/kernel, reusable for a future, larger specialist candidate without redoing the ROCm fix.
  • Cleanup done again, 2026-07-21, after the single-model run above: api.openai_model reverted to deepdelve-gpt-oss:latest/http://localhost:11434/v1 (confirmed via config diff), the config backup at ~/.deepdelve/config.yaml.bak_pre_minicpm_singlemodel_20260721 can be deleted once this entry is read, the vLLM server process (port 8000) killed and confirmed via rocm-smi back to near-zero VRAM use.

• Qwen3-family think-mode control confirmed broken on Ollama too, 2026-07-21 — every Qwen3 benchmark row in README.md's model table was very likely reasoning-polluted. Surfaced while answering the user's direct question ("could the models we benchmarked have a nothink mode too?") after the MiniCPM5-1B finding above. Tested live against Ollama 0.31.2, both mechanisms DeepDelve could plausibly rely on:

  • chat_template_kwargs.enable_thinking: false via the OpenAI-compat endpoint (the mechanism orchestrator.py::_get_default_options() actually sends): confirmed via direct curl against deepdelve-qwen3-4b that this has zero effect — the model still burns its full token budget on unrequested reasoning (a populated reasoning field, content left empty on a 200-token cap).
  • Ollama's own native /api/chat "think": false field (the mechanism Ollama itself recommends for hybrid-reasoning models, and which DeepDelve does NOT currently send at all): confirmed via direct curl against the plain, unmodified qwen3:4b base tag that this is actively worse than doing nothing. With think: false, the model still reasons at length but the raw, unstructured chain-of-thought is dumped straight into message.content with no <think> tag and no separate thinking field at all. With think: true, the exact same request correctly separates reasoning into its own field and content holds only the clean final answer ("4"). The "off" setting is the one that pollutes the model's real working output; "on" is the one that's clean.
  • Why this doesn't apply to gpt-oss:20b (the current default): tested the same two mechanisms against deepdelve-gpt-oss — also ineffective at fully suppressing reasoning (gpt-oss's harmony format always produces an analysis channel by design, this isn't a bug), but critically, Ollama keeps that reasoning cleanly separated into its own reasoning/thinking field in BOTH cases, never mixed into content. Confirmed via agent_framework's own client source (choice.message.content read directly at the point a Content.from_text(...) is built; reasoning_details handled as a distinct text_reasoning content type, never merged into the text DeepDelve's agents treat as the model's actual output) that DeepDelve only ever consumes .content — so gpt-oss's inability to fully disable thinking is benign here, while Qwen3's content-pollution bug is not.
  • Implication: qwen3.6 (35b-a3b), qwen3:4b, qwen3:8b, and the qwen3:4b GRPO fine-tune's live Ollama benchmark run (its TRAINING pipeline correctly used enable_thinking=False via HF's own apply_chat_template, unaffected — see the training entry below — this is specifically about the live benchmark's inference path) were almost certainly running with large amounts of uncontrolled reasoning text bleeding directly into every tool-call argument and piece of written output across their entire benchmarked runs, this whole time. This is a real, previously-unknown contributing factor to their disqualifying failure modes (thin_coverage stalls, narrated-instead- of-written reports, canned non-responses on the corrective nudge) — plausibly consistent with "a small model getting confused/derailed by its own unmanaged internal monologue," layered on top of (not a replacement for) the capacity-floor literature evidence already cited in README.md.
  • Not yet re-tested and not re-scored: no Qwen3 candidate has been re-run with genuine nothink mode (would need the same vLLM/SGLang fix class used for MiniCPM5-1B — ~/.venvs/vllm is already available for this). Existing scores are left standing as the best evidence so far, not silently trusted as clean; README.md's model table now flags every affected row with a † and an explanation rather than treating the old numbers as unaffected. Whether re-testing is worth the time (these are all still sub-14B, below the literature's own capacity floor regardless) is an open call, not yet made.
  • Retrospective audit closed, 2026-07-28: confirmed via a structural scan of all 158 research_output/ run folders (searching every _run_state.json for the run_state.attempt = 10**6 force-final sentinel, the reliable persisted fingerprint of a context-budget/ max_run_minutes forced cutoff — raw stdout logs from past sessions aren't preserved, so this is the only reliable retroactive signal) that no OTHER disqualified candidate beyond the Qwen3-family rows already †-marked above shows this signature. 9 total hits: 2 from the 2026-07-28 Ornith/ Qwen3-4B-LoRA session (already documented, RESEARCH.md §13/§14), 5 unrelated dev/smoke-test runs (trivial queries testing the completion-check mechanism itself, not model quality — see session_status/2026-07-14i.md), and the remaining 2 are these exact same 2026-07-21 Qwen3 investigation runs. mistral-nemo/llama3-groq-tool-use/llama3.2:3b/Bonsai-8B/Gemma/MiniCPM variants show no trace of this signature, consistent with their disqualifications being unrelated failure classes (malformed JSON, wrong tool-call format, citation fabrication from training knowledge) that this specific mechanism can't explain.
  • The "not yet re-tested" blocker above is resolved, 2026-07-28: at the time this open call was left, the only known fix was a full vLLM serving-stack swap — a big step, since reverted (see "Ollama restored" entry below). api.backend: "ollama" (ARCHITECTURE.md §6, added same day) gives the same clean nothink behavior directly through Ollama's own native /api/chat endpoint, live-verified for gpt-oss and Ornith-1.0-9B — no backend swap required. Re-testing any †-marked Qwen3 candidate through this path is now genuinely low-friction; still not actually done, still the user's call on priority, but no longer gated on a bigger infrastructure decision.
  • Confirmed via vLLM, 2026-07-21: the bug is Ollama-specific, not a Qwen3 model limitation. Unloaded gpt-oss from Ollama first (ollama stop, freed ~14.3GB, matching the earlier lesson about not squeezing vLLM into leftover VRAM), launched vllm serve Qwen/Qwen3-4B --tool-call- parser hermes --enforce-eager --gpu-memory-utilization 0.85 --max-model-len 16384 (first attempt at 0.55 under-budgeted the KV cache and failed cleanly with a clear ValueError, not a crash — raised to 0.85, succeeded). Direct curl against the real vLLM server (genuine jinja chat-template evaluation, same class of fix as MiniCPM5-1B):
    • chat_template_kwargs.enable_thinking: false → clean "4.", reasoning: null, 3 completion tokens, zero <think> content anywhere.
    • Same request with enable_thinking: true → full <think>...reasoning...</think> block inline in content (Qwen3's own convention keeps it in content, unlike gpt-oss's separate channel — confirmed as the model's real, correct behavior, not a bug).
    • A real tools= request with enable_thinking: false → clean OpenAI-shaped tool_calls (web_search({"query": "population of Tokyo"})), no reasoning leakage, no stray text. Conclusion: Qwen3-4B's nothink mode is real and works correctly end-to-end once served by something that actually evaluates its chat template — Ollama's failure to do so (confirmed earlier in this same entry) is entirely Ollama's own gap, not evidence against the model. This makes a genuine, clean re-benchmark of the Qwen3 family (via vLLM, same infra now proven twice) a real, low-friction option if it's ever worth revisiting — test server stopped after verification, nothing left running.

• MiniCPM3-4B scoped and attempted as a single-model candidate, 2026-07-21 — INCONCLUSIVE, a real infrastructure hang, not a capability verdict. After MiniCPM5-1B's disqualification, checked other real MiniCPM-family candidates. MiniCPM4-8B/MiniCPM4.1-8B ruled out immediately — their own model cards document no function-calling support at all (only MiniCPM4-MCP, already discarded, was OpenBMB's dedicated tool-use variant of that generation). MiniCPM3-4B looked genuinely promising: documented BFCL v2 71.6 (beats several 7-9B models), Apache-2.0, native vLLM model support (MiniCPM3ForCausalLM). Initially concluded (wrongly, corrected by the user — see feedback_read_docs_before_building.md) that no vLLM tool-call-parser existed for its custom <|tool_call_start|>/Python-function-call format, having only checked the locally installed vLLM package's bundled parsers. OpenBMB's own github.com/OpenBMB/MiniCPM repo (demo/minicpm3/function_call/) ships a ready minicpm_tool_parser.py + matching jinja chat template for exactly this — needed two small compatibility fixes for this vLLM version (0.25.1): import paths moved (vllm.entrypoints.openai.protocol → .chat_completion.protocol + .engine.protocol; vllm.entrypoints.openai.tool_parsers → vllm.tool_parsers), and the base ToolParser.__init__ now takes a second tools param the reference script's subclass didn't accept. Confirmed working after patching: real structured tool_calls out of a direct curl test, no narrated JSON.

  • Real hardware ceiling found, applied correctly THIS time before benchmarking: MiniCPM3-4B's 62-layer, non-MLA-optimized-in-this-config KV cache cost forced a real serving ceiling of ~6144 tokens on this GPU (vLLM's own KV-cache-budget error gave this number directly), well under the project's ~16K-token floor (context_budget_chars: 50000's documented "safe margin under a 16K-token num_ctx"). First response was to proportionally scale context_budget_chars down to 8000 and run the benchmark anyway — the user corrected this as the wrong general policy going forward (new Model Evaluation Standard point 6, above): a candidate that can't clear ~16K tokens should be discarded outright on hardware grounds, not accommodated by rescaling the project's own safety margins. This specific run was allowed to finish since it was already informative either way, but is not the template for future candidates.
  • Result: a real hang, not a clean pass or fail. The DeepDelve run itself showed zero visible progress for ~16+ minutes past the startup banner. Diagnosis: vLLM's own periodic engine-stats logger (normally prints every ~10s) went completely silent after the first exchange, the APIServer process (not EngineCore) was pinned at ~94% CPU while GPU utilization sat at only 7%, and even the lightest possible request (GET /v1/models) timed out entirely. This pattern points at OpenBMB's own reference extract_tool_calls_streaming — it re-scans the ENTIRE accumulated generation text with a nested-parentheses regex (r"(\w+)\(((?:[^()]*|\([^()]*\))*)\)") on every single streamed token, a known catastrophic- backtracking risk class, not something DeepDelve's own code touches. Killed the hung run and server rather than let it burn GPU time indefinitely; confirmed no leftover VLLM::EngineCore process afterward (this evaluation's third time hitting that exact leftover-process gotcha — always rocm-smi --showpids after any pkill/kill of a vllm serve parent, the EngineCore child does not reliably die with it).
  • Verdict: NOT a capability disqualification like MiniCPM5-1B's — this never reached the point of testing MiniCPM3-4B's actual research/delegation behavior at all, so per the Model Evaluation Standard's point 1 (confirm the operating mode works before scoring), this doesn't count as a settled discard. It's an open infrastructure question: OpenBMB's own reference tool-parser has an apparent streaming-performance bug (or this vLLM version's streaming invocation pattern doesn't suit it) that would need a real fix (e.g., incremental parsing instead of re-scanning full text per token) before a fair benchmark could run. Not pursued further this session — flagged as genuinely unresolved, not "MiniCPM3-4B discarded."

• MiniCPM4-MCP evaluated as a specialist-role candidate, 2026-07-20 — real infrastructure built and kept, model itself not yet viable. User surfaced github.com/openbmb/minicpm; downloaded MiniCPM4-MCP (the tool-use SFT checkpoint, not the base chat model — see RESEARCH.md's §6.2 entry on why the base checkpoint doesn't inherit the MCP fine-tune's tool-calling numbers), Q5_K_M GGUF via ollama pull hf.co/mradermacher/MiniCPM4-MCP-GGUF:Q5_K_M (5.8GB, comfortable on 16GB VRAM), local tag minicpm4-mcp with num_ctx set to the model's real native max (32768, confirmed via the GGUF's own minicpm.context_length metadata and the upstream config.json's max_position_embeddings — going further to the maker's documented 128K-validated LongRoPE factors would require re-converting the GGUF from patched source weights, not just an Ollama parameter, deferred for later).

  • Format mismatch found and solved: MiniCPM4-MCP's own embedded chat template doesn't emit OpenAI-style JSON tool_calls — it emits a <|thought_start|>...<|tool_call_start|> func(arg=val)<|tool_call_end|> Python-code-block format. Ollama's generic /v1/chat/ completions tool-calling support assumes OpenAI JSON and fails outright against this model ("peg-native format" 500 error, confirmed live). Built finetune/minicpm_tool_proxy.py: a FastAPI translation proxy (checked GitHub for prior art first — philipluo/MY-LITE-LLM does the same class of thing generically for minicpm-v; this one is tailored to MiniCPM4-MCP's actual documented format instead of generic JSON-prompting) that builds the model's own "# Functions" prompt block from OpenAI tools= schema, renders full multi-turn history (including prior tool_calls/tool-result messages) into the model's native turn format, and parses its Python-code-block output back into OpenAI-shaped tool_calls JSON. Verified in isolation: single-turn tool call, multi-turn tool-result round-trip (model correctly answered directly instead of re-calling once given a result), both correct.
  • New config plumbing added to make this pluggable: settings.specialist_base_url (src/tools/config_template.yaml, src/engine/orchestrator.py's _build_client) — an escape hatch alongside the existing settings.specialist_model for a specialist model that needs a DIFFERENT endpoint (the translation proxy), not just a different model name on the same endpoint. Real bug caught and fixed while wiring this in: _build_client's injected AsyncOpenAI(base_url=...) — the object that actually issues HTTP requests, not the wrapper OpenAIChatCompletionClient — was still hardcoded to api_cfg["openai_base_url"] even after adding the override parameter, so the first live-test attempt silently bypassed the proxy entirely and hit Ollama directly (same "peg-native format" error as before this whole effort). Fixed; test_structural_checks.py and ruff check both clean after.
  • Live end-to-end result, real query, real pipeline (same Rust-version + borrow-checker query used throughout the routing-classifier verification above): the fix held — proxy received real traffic, tool calls flowed correctly in both directions, run completed with a real report (not a crash, not a silent drop). One genuine positive: this run's AcademicSearcher/DocumentAnalyzer chain (via MiniCPM) surfaced a real academic source (ETH Zürich "Implementing a Sound Borrow-Checker") that the earlier gpt-oss run never found, alongside the same arXiv LLBC paper both runs found.
  • A real, distinctive new failure mode also surfaced, not predicted by the isolated tool-call test: a nested DocumentAnalyzer sub-agent (routed through MiniCPM, since Analyzer roles share the specialist tier) called read_workspace_file/grep_workspace_file/ extract_structured_data with filename: "Analyze paper metadata" — ITS OWN TASK LABEL, not a real file — repeatedly, never correcting after identical "not found" errors each time, until the sub-agent was re-dispatched as a fresh instance 10 separate times. Task-name/ filename confusion with no self-correction, a new category distinct from anything the earlier routing-classifier or grounding-check work targeted.
  • Reliability was meaningfully worse than the current tier under real load: 53 tool errors this run vs. 0-8 in clean deepdelve-gpt-oss baseline runs, ~900s runtime vs. ~680-810s, 4 BuilderFix + 4 ReviewFix remediation cycles to clear an uncited_claims check (Builder itself still runs on the main model, so this is downstream noise from messier findings content feeding it, not MiniCPM's tool-calling directly — but a real cost of using it anyway).
  • Verdict: the translation-proxy infrastructure is sound and kept as a real, reusable project artifact — genuinely solves the format-mismatch problem for any future MiniCPM-family (or similarly non-OpenAI-native) candidate. MiniCPM4-MCP itself is not yet a viable specialist-role candidate — directly the same standing lesson this project has hit repeatedly: an isolated tool-call test passing does not predict live multi-agent-role reliability. Live config's specialist_model/specialist_base_url reverted to unset (back to the known-good single-model baseline) after this evaluation.
  • Not done, deferred: re-converting a GGUF with the maker's 128K-validated LongRoPE factors (32K is architecturally native/what's baked into the current GGUF, not an Ollama-imposed ceiling — see the maker's own README) — user wants to revisit 128K-context options generally later, not specific to MiniCPM.
  • CORRECTION, same day: the verdict above was reached before reading OpenBMB's own reference implementation (demo/minicpm4/MCP/generate_example.py + model-card usage docs) — user caught this explicitly ("I told you to search implementations and you did the development believing you're a bad ass, don't do the mistake again, if it's new we need to read documentation"). Reading it afterward surfaced two real, concrete gaps in the proxy, not assumptions: (1) OpenBMB's own reference system prompt has explicit anti-repeat-tool-call guidance ("If a tool fails... DO NOT call it again with the same inputs... avoid redundant or circular behavior") that this proxy's system prompt never included; (2) their reference parser (parse_tool_for_minicpm3) handles Python-keyword-colliding argument names and hyphenated tool/argument names (real MCP tool-naming conventions) via a temp-rename round-trip that this proxy's simpler regex+ast.literal_eval parser silently dropped. Confirmed their own raw-prompt-plus-custom-parser integration pattern (client.completions.create with a tokenizer.apply_chat_template-rendered prompt, not the chat/tools API) validates this proxy's core architecture, though — not a wrong approach, an incomplete one.
    • Both gaps fixed in finetune/minicpm_tool_proxy.py: added the anti-repeat guidance verbatim to build_functions_preamble; replaced the parser with an AST-module-body walk (parse_tool_call_block) ported from their parse_tool_for_minicpm3/ resolve_ast_call/resolve_ast_by_type, handling keyword-collision and hyphen round-tripping the same way. Verified in isolation: search_papers(from="2020", to="2024") and get-weather(city="London") — both previously silent parse failures — now parse correctly.
    • THIRD live test, same query, with both fixes: the SPECIFIC bug this was meant to fix (task-name-as-filename looping) did NOT recur — confirmed gone. But the run surfaced DIFFERENT reliability problems in its place: web_search/fetch_url_to_workspace quota exhausted (17 calls against a 15 limit, excessive re-querying rather than converging); the existing topical_mismatch completion check caught the draft report citing a Yahoo Sports article and an unrelated tech listicle as "Rust" sources (noisy search, safety net worked, but reveals messy upstream search behavior); and the final report itself regressed in accuracy versus the earlier successful run — cited a blog aggregator (emergentmind.com) instead of the real peer-reviewed arXiv paper the second run found correctly, and reported Rust 1.97.0 as current when 1.97.1 (confirmed correct in earlier runs) is the actual latest patch.
    • Revised, still-honest verdict: the doc-informed fixes solved the exact bug they targeted, but MiniCPM4-MCP's reliability in this real multi-step research role remains inconsistent run-to-run — one problem fixed, two different problems surfaced in its place. Still not a stable specialist-role candidate as of this evaluation. Live config's specialist_model/specialist_base_url reverted to unset again; proxy process stopped.

Tongyi-DeepResearch-30B-A3B candidate — two real benchmark attempts, both impractical (moved from the reference-repo candidates list during the 2026-07-21 status audit: this was tried, twice, not merely proposed):

• Tongyi-DeepResearch-30B-A3B as a benchmark candidate (from Alibaba-NLP/DeepResearch): 30B MoE / 3.3B active — same size class as deepdelve-qwen3.6, but trained specifically for long-horizon research. Architecture, read directly from the primary paper (arXiv:2510.24701) during the comparative survey, RESEARCH.md §7, 2026-07-20: this is a SINGLE fine-tuned model operating via ReAct or an "IterResearch"-based Heavy test-time-scaling mode — not a multi-agent system in DeepDelve's sense at all (no Planner delegating to typed specialists with independent context). No published runtime grounding/citation-verification layer comparable to DeepDelve's own — reliability, to the extent it's addressed, comes from the training pipeline (continual agentic pre-training + on-policy GRPO) rather than a deployment-time safeguard. Backed by an 18-paper research program (WebWalker, WebDancer, WebSailor, WebShaper, WebResearcher, and more) — a frontier-lab-scale effort DeepDelve isn't attempting to match; if adopted, it would be solving DeepDelve's reliability gap by swapping in a much larger purpose-trained model rather than by DeepDelve's own verification-layer approach, and would still need DeepDelve's own grounding checks layered on top if citation-level provenance matters for the use case (Tongyi's benchmarks measure answer-correctness, not per-citation provenance the way DeepDelve's own checks do). Chat-template/tool-call compatibility check done, 2026-07-12 — the flagged risk is resolved: deepdelve-tongyi (built pre-outage from hf.co/mradermacher/Tongyi-DeepResearch-30B-A3B-GGUF:Q4_K_M, 18.6GB, num_ctx 16384) reports Ollama capabilities ['completion', 'tools', 'thinking'] — the community GGUF's chat template parses the model's native <tool_call> XML into real structured tool_calls (verified live via a direct /api/chat call with a tool schema: returned a proper tool_calls array, not raw XML text). A real --depth quick trial run (compare_the_vector_search_capabilities_of_elastics_...) confirmed delegate_tasks actually gets invoked with 2 real specialist tasks, 2 real fetches, and write_todos populated correctly — passing the exact bar devstral:24b failed (README "Model choice": zero real delegate_tasks calls, narrated JSON instead). The run didn't finish within a 5-minute smoke-test window — Tongyi's <think> traces are verbose (one single-tool-call test round-tripped a 1000+ token thinking block for "15 + 27") — so a real benchmark round needs a longer time budget than the other local candidates, not a template fix. Config for testing: ~/.deepdelve/config-tongyi.yaml (not in git, mirrors the live config with openai_model: deepdelve-tongyi).
  • Two real benchmark attempts, both inconclusive on quality — the model is not currently usable at either quant tried, for two different reasons. Q4_K_M: killed at 1h6min (the max_run_minutes bug this exposed and fixed, see "Repo governance + CI" entry above) — GPU was genuinely computing the whole time, real progress happened (delegate_tasks invoked, 2 fetches), just far too slow to be practical. Then tried deepdelve-tongyi-iq3 (hf.co/mradermacher/Tongyi-DeepResearch-30B-A3B-i1-GGUF:IQ3_M, 13.5GB — passed the same isolated tool-call smoke test, and was noticeably faster/less verbose on that trivial test: 2.7s vs. 5.9s eval time for "15+27") expecting it to be the practical answer. It was worse on the real workload: 37+ minutes against the actual Planner system prompt with ZERO progress — no write_todos, no delegate_tasks, no run folder content at all (_run_state.json stayed at its initialized empty state the whole time), unlike Q4_K_M which at least made real tool calls in a comparable window. Killed manually. The isolated single-tool-call smoke test (README's curl .../api/chat snippet) evidently does NOT predict real-workload viability at this quant level — a real trial against the actual multi-thousand-token Planner prompt is the only test that means anything, and neither quant has passed one yet. Not recommended for further local benchmarking without a materially different quant or a context/prompt-length investigation into why the full system prompt specifically breaks it.