Changelog Recent

Changelog: Recent (2026-07-26 through 2026-08-17)

Part 1 of the Changelog — most recent entries first.

2026-08-17 (latest): Seven real bugs found and fixed across 6 live runs, closing the "per-source evidence-crowding" item that stood as the #1 priority since 2026-07-22

Closed out every open item from session_status/2026-08-16.md, then kept live-testing the same "Lisbon vs Mexico City" prompt through 4 more full runs, root-causing each new failure via direct _run_state.json/session-transcript reads rather than trusting log lines alone. Full traces in ARCHITECTURE.md §2/§3's updated landmine writeups; summary here, roughly in the order fixed:

1. RunState.coverage() empty-summary exclusion (commit 7eefe5e) — a real, http-prefixed source_url used to count as "covered" even when its summary was completely empty (a sub-agent ending its own turn immediately after a tool call, zero trailing text, no cutoff marker at all — a third, previously-undetected synthesis-vanishing mechanism, measured at 25%/42% of all findings across two live runs). coverage() now excludes via the same _is_null_finding_summary predicate used elsewhere, fixing check_thin_coverage/check_uneven_task_investment automatically (both consume coverage()'s output directly).
2. Ledger rollup + FOLLOW-UP DIRECTIONS grounding fix (commit 87b2d73) — a depth==1 task whose OWN findings were all empty still had real evidence in its depth>1 Analyzer children that _update_task_verification never credited back to it; new top_level_task_name contextvar fixes the rollup. Separately, a specialist's FOLLOW-UP DIRECTIONS: section (suggested next URLs, never a citation) was firing false SYSTEM VERIFICATION WARNINGs that invalidated genuinely-cited real content next to it.
3. Task-name-churn denominator inflation fix (commit 182d2e6) — _looks_like_renamed_task's raw difflib char-ratio scored 0.11 on a real live full-sentence paraphrase (nowhere near its 0.6 threshold), so a facet redispatched 3 times under 3 different names was never recognized as one rename — and even when caught, coverage() never read the superseded ledger status, so the denominator kept inflating anyway. Fixed both: a content-word-overlap OR-trigger, and coverage() excluding superseded task_names.
4. URL-scoped verification-warning exclusion (commit 4c5a4c3) — root cause of the "per-source evidence-crowding" item first named 2026-07-22 and repeatedly flagged since as the clear #1 priority (session_status/2026-08-16.md item 3): add_finding attaches ONE shared synthesis text to every URL fetched in a turn, so a stub_source/unverified_urls flag about ONE co-fetched URL wholesale-excluded the record for ALL of them — confirmed live, a real MX$17,300/month price got thrown away alongside an unrelated stub-page flag. _is_citable_finding now scopes the exclusion to the finding's own source_url when the marker names specific bad URL(s).
5. read_workspace_file exact-repeat quota dedup (commit b442797) — a FindingsWriter dispatch called read_workspace_file with IDENTICAL arguments 2-3 times in a row, burning its entire quota before finishing its actual edit work (confirmed: "41/47" quota-exhaustion, forcing a final_report.md-never-written outcome). Live-confirmed working: zero quota-exhaustion events in the very next live run, versus the prior run's clear 41/47.
6. Warning-marker leak fix (commit ba8dddf) — a direct side effect of fix 4, found in the NEXT live run after it shipped: a finding correctly staying citable (its own URL wasn't the flagged one) still carried the raw warning marker TEXT verbatim in its summary, which got rendered into findings.md by the deterministic fallback and then re-flagged by findings.md's own grounding check as if the warning's named URL were a real citation — a self-inflicted loop, confirmed via 3 byte-identical findings.md.rejected_attempt_N snapshots. Fixed by stripping the marker text before rendering a citable finding's block.

Two real issues found but deliberately left open, not rushed: (a) a writer-role dispatch (FindingsWriter) can end its own turn with zero output after a writer_gate_ctx block — the same "zero trailing text" mechanism as fix 1, now confirmed to also hit a writer role, where the consequence is worse (nothing written at all); the existing one-shot retry only partially absorbs it. (b) why FindingsWriter's rebuild sometimes fails to self-correct a flagged citation across multiple retries. Both need their own properly-scoped investigation, not a tail-end patch — see session_status/CURRENT.md for the live-run evidence backing each.

2026-08-16: Four real completion-check bugs found and fixed by repeatedly live-testing one prompt instead of trusting a single pass

Ran the medium "Lisbon vs Mexico City" eval prompt (eval/dataset.jsonl) as a smoke test for an unrelated open item (_find_sibling_fetch), and instead of getting a clean confirmation, kept hitting real, DIFFERENT bugs each of 5 consecutive live runs (~45-68 min each). Chased each one down to its actual root cause via _run_state.json/findings.md/final_report.md (not log lines alone — a mid-session user correction, "you're too confident," caught a case where the run's "System" log messages looked like healthy progress while the actual report on disk was still badly incomplete) rather than retrying blind. Full root-cause traces in session_status/2026-08-16.md and ARCHITECTURE.md §1/§2's own updated landmine writeups; summary here:

1. gap_acknowledged directive-oscillation fix (commit cdf354c) — check_task_verification_flagged's quota-exhausted "stop redelegating" directive could get silently reversed by a LATER retry_quota_topup refill flipping quota_exhausted back to False, reissuing "redelegate" for a task already told to stop. gap_acknowledged on the per-task ledger entry now makes that decision sticky across the ledger's full recompute.
2. Cross-tier completion-check starvation fix (commit 1e70bba) — the existing starvation guard only protected GROUNDING_CHECKS when the SAME COMPLETION_CHECKS problem repeated consecutively; a run where a DIFFERENT COMPLETION_CHECKS problem won every attempt starved GROUNDING_CHECKS (specifically report_underuses_evidence, built to catch exactly this) for an entire run despite final_report.md having dropped 3 of 4 requested facets. New _COMPLETION_TIER_PROBLEMS/_consecutive_tier_wins generalize the guard to the whole tier. Live-confirmed working: post-fix, the same check correctly kept firing and dispatching Builder across 11 attempts instead of never getting a turn.
3. writer_gate_ctx / edit_workspace_file fix (commit 1e70bba) — the structural gate forcing FindingsWriter to write before reading only accepted write_workspace_file, but is armed for the per-facet ADD-ONLY correction dispatches too, whose own instructions say to use edit_workspace_file instead and never touch the rest of the file. The gate's block (wording: "call write_workspace_file now") actively steered the model toward exactly the full-file overwrite its own instructions forbade, silently destroying facets a prior round had already added. Fired 20 times in one run that never converged. Live-confirmed working: zero occurrences post-fix, findings.md converged cleanly on the first pass.
4. _looks_like_renamed_task entity-mismatch fix (commit 1e70bba) — the heuristic meant to catch the Planner renaming a flagged task instead of retrying it under the same name false-positived on two INDEPENDENTLY dispatched, differently-named tasks sharing a template (two cities' rent facets differing only in city/neighborhood names, 0.89 raw difflib similarity). This silently, permanently marked one city's real facet "superseded" the moment the other verified — it never appeared in findings.md or the final report, with no gap ever flagged anywhere in the run. Fixed via proper-noun extraction + a Jaccard-overlap override that only trusts a high text-similarity ratio when the two tasks' actual named subjects overlap. Found too late in the session for a fresh live run to finish before this write-up — unit-tested, live reconfirmation still open (see session_status/CURRENT.md).

Related, found but deliberately NOT fixed this session (own future investigation): a single task that legitimately fetches many real sources (8+ seen live) can still crowd FindingsWriter's one-shot synthesis even after the existing per-facet dispatch fix (which scopes by TASK, not by individual source) — confirmed live, 11 of 13 source headings in one findings.md got a placeholder despite real content existing for at least one of them, and Builder then fabricated plausible numbers for those gapped sources in the final report. See session_status/CURRENT.md item 2.

2026-08-04: First hosted frontier-model test (deepseek-v4-flash/deepseek-v4-pro) — DISQUALIFIED, but two real cross-cutting harness bugs found and fixed along the way

First-ever hosted (non-local) model tested against this project's real pipeline, via src/api.py + web UI and the headless eval/evaluate.py harness. 4 total runs (2x flash via web UI, 1x flash headless CLI, 1x pro headless CLI) against eval/dataset.jsonl's complex-difficulty Colombia B2B item — DeepSeek's own thinking-mode/tool-calling docs (api-docs.deepseek.com/guides/thinking_mode, /guides/tool_calls) were read directly before any config change, not assumed from memory, after an initial wrong guess at DeepSeek's current model names (deepseek-chat/deepseek-reasoner, both stale — current models are deepseek-v4-flash/deepseek-v4-pro) was caught and corrected via the real pricing docs first.

Two real, reusable harness bugs found and fixed, both worth more than this one model's own verdict:

1. api.backend: "openai_hosted" (src/engine/orchestrator.py's _get_default_options + new _HOSTED_PROVIDER_THINKING_EXTRA_BODY provider map, src/config.py's new OPENAI_API_BACKEND env overlay): the existing "openai" backend's thinking-mode control (chat_template_kwargs + reasoning_effort:"none") is a vLLM/local-serving convention a real hosted API just silently ignores — confirmed live that DeepSeek defaulted to thinking ON at effort high regardless, and that chat_template_kwargs is meaningless to its API at all. "openai_hosted" looks up each hosted provider's own documented convention instead, keyed by openai_base_url. Confirmed fixed via _agent_session.json showing zero reasoning_content occurrences post-change — but this alone did NOT fix DeepSeek's failure, isolating that its narration is verbose-as-content, not leaked hidden reasoning.
2. check_task_verification_flagged guardrail-starvation bug in src/api.py (its context_budget_chars cutoff, was an unconditional run_state.attempt = 10**6 force-jump): confirmed live that a verbose model can blow the same 50000-char budget gpt-oss:20b never approaches, before the run's FIRST completion-check attempt even happens — giving a check that was actively working (verified-task count improving run-over-run) literally zero real retries before salvage. api.py's own code comment had explicitly, deliberately chosen this blunter behavior over run_cli's existing two-stage nudge-then-cutoff for shared-queue-protection reasons; brought to parity anyway since one bounded extra wrap-up turn doesn't meaningfully weaken that protection and the alternative (a guardrail that can never fire) is worse. Verified the fix works: a subsequent run's completion_check_attempts showed real numbered attempts (0,1,2,3) with task_verification_flagged correctly escalating and yielding to missing_findings per its own documented 3-strike design, instead of jumping straight to the 1_000_000 sentinel with zero attempts like both pre-fix runs did.

With both fixed, DeepSeek still failed — root cause is genuine model unreliability. Score: 0.2/1.0 (flash), 0.0/1.0 (pro, worse despite ~3x the price) vs. gpt-oss:20b's 0.7/1.0 baseline, same rubric. The real, confirmed cause: DeepSeek re-fabricates the same citations on redo — check_task_verification_flagged correctly refused to let the run advance to the check that dispatches the real FindingsWriter/Builder writer roles, because the SAME flagged task names recurred across retry attempts instead of resolving (confirmed via subagent_invocations: zero Builder/FindingsWriter dispatches in the run that reached this state). Not disqualified for: tool-calling mechanics (clean in 3/4 runs) or "no write access" confusion (the Planner role genuinely has no write_workspace_file tool by design — DeepSeek's own statement to that effect was accurate, not hallucinated). Full verdict and evidence trail in MODELS.md's new "Hosted" section entry.

2026-07-29: "no proper report despite solid findings" investigated end-to-end — one real coverage gap fixed, one real check-priority-shadowing bug fixed, both live-verified

User pushed back on the earlier 2026-07-29 completion-check cleanup session's own diagnosis that the findings/report-writing stage wasn't structurally overwhelming models, insisting on a fresh, whole-repo pass over every historical run rather than trusting the prior audit's conclusion. A dedicated catalog pass over ROADMAP.md's full History, RESEARCH.md, and every session_status/ *.md archive found 17 distinct report-writing-stage incidents across 5 patterns, spanning 2026-07-10 through 2026-07-28 and every model this project has ever tested, including the trusted gpt-oss:20b baseline. Two real, distinct root causes were fixed as a result — see Completed below (check_report_underuses_evidence, the check-priority-shadowing fix in completion.py/ grounding.py) — both grounded in checked literature (NAACL 2025 "Coverage-based Fairness in Multi-document Summarization", arXiv:2412.08795; "Regression Accumulation in Multi-Turn LLM Programming Conversations", arXiv:2607.01855) and both live-verified: the first via a real 48-minute gpt-oss:20b run against the standing 2-facet benchmark query, the second by feeding that SAME run's actual saved final_report.md into the fixed code directly and confirming the previously- silent uncited_claims problem now surfaces.

A live smoke test caught a real gap in my own verification, worth recording: the first "smoke test passed" read of that 48-minute run was wrong — it checked for crashes and false-positive regressions on the new check, but never read what the report actually said. The user rejected that read outright ("this is not acceptable, no proper report made") and was right to: the report used a fabricated (Source 1)...(Source 8) inline numbering scheme with no numbered reference list at the bottom (unverifiable to a human despite passing URL-presence grounding), a claim attributing payroll-tax coordination to a source that, read directly, never mentions it, and — the concrete bug fixed this session — check_uncited_claims never got a turn across 3 attempts because check_stub_source kept winning real_grounding_problem's own first-hit priority chain, and the terminal "retry budget exhausted" message reported only stub_source even though uncited_claims was independently, simultaneously true. Read the real output, not just the check log, before calling any smoke test conclusive.

2026-07-28: Ornith-1.0-9B bake-off — INCONCLUSIVE, two real DeepDelve architecture bugs found+fixed, one serving-layer gap isolated

Full trail in RESEARCH.md §14 (a-g); working summary in session_status/CURRENT.md. Candidate: deepreinforce-ai/Ornith-1.0-9B (dense, Qwen3.5-arch, MIT), the untested middle ground identified after gpt-oss:20b between the already-exhausted sub-14B local-model space and paid frontier APIs. GLM-4.7-Flash and Ornith-1.0-35B were both ruled out on hardware grounds first (smallest available quants 19GB/21.2GB, over this hardware's 17.1GB VRAM budget) without spending GPU time on either.

Five live runs (cold pull + benchmark, then three --resume-run attempts) surfaced a genuinely strong cold-start synthesis (45 real sources, correct architecture-family coverage matching the benchmark's own gold reference — the best findings.md of any candidate this project has tested), but never converged on a clean, fully-verified final_report.md. Root cause of the looping/ self-rejection pattern traced to the stock chat template's empty-<think>-injection defect, independently corroborated by deepreinforce-ai/Ornith-1's own GitHub issues (#4, #16) and two Reddit threads (r/LocalLLaMA) — a real, model-family-wide trait, not a DeepDelve-specific one. Fixed via froggeric/Qwen-Fixed-Chat-Templates, patched directly into the pulled GGUF's tokenizer.chat_template metadata (gguf_new_metadata.py --chat-template-file, no tensor rewrite).

Two real, model-independent DeepDelve architecture bugs found and fixed along the way (would affect any model hitting these paths, not specific to this candidate):

1. check_not_delegated was scoped to the current process's live quota pool (always 0 at the start of a resumed process), contradicting build_resume_input's own "don't re-delegate" instruction — live-confirmed to derail a resumed run into a think_tool reflection loop. Fixed: Ctx.delegated now also checks run_state.data["fetched_urls"] (already resume-carried-over ground truth). ARCHITECTURE.md §4 updated from "known, not fixed" to "fixed".
2. Builder was dispatched with grounding-check inject text worded for the Planner ("delegate a Searcher"), but Builder has no delegate_tasks tool at all — live-confirmed to get it stuck narrating "I will delegate..." instead of ever rewriting the file. Fixed: a shared _BUILDER_NO_DELEGATE_CLARIFICATION string appended to both Builder-dispatch branches. ARCHITECTURE.md §2 gained a new subsection on this class of bug.

Both fixes have regression tests in test_structural_checks.py; full suite passing.

Added edit_workspace_file (src/tools/fs.py, targeted old-string/new-string replacement, wired into both Builder and FindingsWriter — app.py, quota in config_template.yaml + live config, prompt text in prompts.py) after confirming live that a "drop 3 flagged citations, keep everything else" correction cycle was a genuine capacity edge for full-document regeneration (one attempt made zero write/edit calls across ~8 minutes; successive full rewrites fixed old stub citations while introducing new ones, never converging). Live-tested once after adding it — the model didn't spontaneously reach for the new tool even on a textbook case; inconclusive on one sample, not contradicted, left open for a future retest.

Real serving-layer gap isolated and confirmed (RESEARCH.md §14e, ARCHITECTURE.md §6, new): four direct API tests holding everything else constant showed thinking suppression works cleanly via Ollama's native /api/chat endpoint, even with tools present, but leaks via the OpenAI-compat /v1/chat/completions endpoint DeepDelve actually uses — specifically triggered by tools being present in the request. Since DeepDelve is built entirely on an OpenAI-compatible client, this likely affects every model ever tested through Ollama on this project whenever it makes a tool call with thinking nominally disabled, not just this candidate. Not fixed — proposed as a real architecture question (a backend-adapter abstraction: how does a serving backend want thinking toggled / tool schemas shaped / errors surfaced, replacing the growing pile of if base_url contains X special cases in _get_default_options()), not yet designed or scoped. Added to Pending below.

Process hygiene lesson: nohup python src/app.py ... & returns the bash wrapper's PID, not the actual Python process's — killing the wrapper alone left two real app.py processes orphaned in the background mid-session, each still holding a request against the model. Use pkill -f "resume-run <folder>" or capture the real child PID directly next time.

Verdict: Ornith-1.0-9B left INCONCLUSIVE, not DISQUALIFIED, not PASSED — every failure mode hit had an independent non-model explanation attached (two fixed DeepDelve bugs, one serving-layer gap, one missing tool now added), so no run tonight was a clean, unconfounded test of this candidate's real ceiling per the Model Evaluation Standard. A future clean re-test (proper process tracking, full attempt budget already in place) is the natural next step, not a repeat of tonight's confounded conditions. Ollama tags (deepdelve-ornith-9b, deepdelve-ornith-9b-froggeric) and the patched GGUF left in place, not cleaned up, since this isn't a closed disqualification.

2026-07-28: Fine-tuning resumed — comprehensive 7-dimension combined GRPO round trained, held-out-evaluated, confirmed real generalization

User explicitly decided to resume fine-tuning after the indefinite pause (see "Stretch" section's own standing note, updated alongside this entry), on the condition that every documented reward dimension be accounted for first rather than training with partial coverage again. Auditing finetune/reward.py against what finetune/train_combined_grpo.py actually trained (vs. what its own docstring merely documented) found 3 real gaps, all closed same-day: real_tool_name_reward (had real calibration data, tool_name.jsonl, but no synthetic-prompt generator, never trained); check_stale_findings and check_uneven_task_investment (no reward function at all). New generators built for all three (generate_synthetic_tool_name_prompts.py, generate_synthetic_stale_findings_prompts.py, generate_synthetic_uneven_investment_prompts.py), plus task_verification_flagged scenarios added to the existing generate_synthetic_prompts.py (today's own live VERIMAP quota-fix subject, reuses thin_coverage_response_reward — confirmed structurally identical response shapes, not assumed). Full design plan at ~/.claude/plans/imperative-tumbling-journal.md.

Training round: Qwen/Qwen3-4B base, LoRA (r=16), 7 task_types / 302 rows / 5 distinct reward functions (up from 3 task_types / 158 rows in the prior 2026-07-19 round) — the "one combined retrain, never piecemeal" methodology rule (below) honored fully this time, not partially like the 2026-07-21 attempt that added writer_role_response_reward but never actually launched.

Took 5 attempts to complete, 2 real bugs found and fixed via live training (both genuine model-output edge cases the reward functions weren't defensive against, not hypothetical):

1. findings_underuses_evidence_response_reward crashed on a None per-task URL list — root cause: datasets.Dataset.from_list unions every row's dict keys into one Arrow struct schema, padding keys absent from a given row with None (not an empty list) since different findings_evidence scenarios name different tasks. Fixed: treat None like [].
2. schema_compliance_reward crashed (AttributeError) when a real completion emitted "arguments" as a JSON array instead of an object — ironic, since this is exactly the malformed shape that function exists to score 0.0 for. Hit at step 250/260 (96% through a ~95-minute run, save_strategy="no" meaning nothing was saved yet — a full restart). Fixed once at the root with a new shared _tool_args() helper (not patched separately at the 4 call sites that had the same unguarded tool_call.get("arguments") or {} idiom). Checkpointing (save_steps=50, save_total_limit=2, --resume flag) added afterward so a future crash costs at most ~50 steps, not the whole run.

Held-out evaluation (finetune/evaluate_combined.py, extended today from 2 dimensions to all 7 — was a real pre-existing gap, findings_evidence had no held-out coverage either before today): 39 held-out prompts, topics never in the 302 training rows.

Dimension	base	fine-tuned	delta
thin_coverage	0.542	1.000	+0.458
citation_grounding	0.615	0.781	+0.167
findings_evidence	0.562	1.000	+0.438
tool_name	0.925	1.000	+0.075
stale_findings	1.000	0.875	-0.125
uneven_task_investment	1.000	1.000	0.000
task_verification_flagged	1.000	1.000	0.000
combined overall	0.747	0.926	+0.179

Verdict: real generalized improvement confirmed, not memorization — the two hardest, most load-bearing dimensions (thin_coverage, findings_evidence) improved substantially on topics the model never trained on, directly answering the overfitting question raised mid-session. Real caveats: stale_findings regressed (-0.125) but on only 2 held-out prompts, plausibly noise; citation_grounding (the dimension most tied to actual hallucination risk) improved but did not reach ceiling (0.781) — the residual gap here matters most of any dimension.

NOT YET DONE — this is still an offline reward-function score, not a live-verified result. Adapter saved to /mnt/nuevovol/llm-models/qwen3-4b-combined-v2-lora (a NEW directory, the prior 2026-07-19 2-dimension LoRA at qwen3-4b-combined-lora was deliberately left untouched pending this round's own validation). Before this counts as deployable: (1) load the LoRA into a real DeepDelve run and benchmark it live, same standard every model candidate is held to, (2) specifically re-test a citation_grounding-shaped scenario given the residual gap, (3) decide whether the stale_findings regression is real (more held-out prompts for that one dimension) or noise. Full session detail in session_status/2026-07-28.md once archived (see CURRENT.md if not yet archived).

2026-07-28 (later): v2 combined LoRA live-benchmarked end-to-end — DISQUALIFIED

Closes the "NOT YET DONE" caveat on the entry directly above. Deployment pipeline: merge_and_unload() (CPU) → GGUF q8_0 via convert_hf_to_gguf.py (4.27GB) → Ollama tag deepdelve-qwen3-4b-combined-v2 (num_ctx 40960, reused the proven qwen3:4b tool-call template) → direct tool-call smoke test passed cleanly → ~/.deepdelve/config.yaml pointed at it → live run against the standing sales-forecasting/heuristic-algorithms benchmark (eval/sales_forecasting_benchmark.md), same query every prior model candidate has been run against.

Result: DISQUALIFIED — final_report.md never written, retry budget exhausted. Run folder: research_output/i_want_documentation_on_heuristic_algoritms_for_de_20260728_143953/. check_task_verification_flagged fired twice against the same 3 tasks for fabricated/unverified citations; the model's own first synthesis attempt narrated report content in prose instead of calling write_workspace_file ("Since I cannot write or edit files directly, I will describe the content..."). After 2 failed grounding-verification retries it re-dispatched a duplicate/untracked task, tripping check_untracked_delegation's hard stop before any report was produced. This directly answers the open "live citation_grounding retest" question from the entry above: still broken live — consistent with the held-out eval never reaching ceiling (0.615→0.781). The stale_findings open question was not exercised (no report to compare).

Model Evaluation Standard point 3 note: confirmed via raw API test, on both this candidate and plain qwen3:4b, that the run executed under the already-known, already-accepted Qwen3 think-mode-passthrough Ollama bug (chat_template_kwargs.enable_thinking:false + reasoning_effort:none suppressed nothing on either tag) — a pre-existing, model-family-wide confound (see the "Ollama restored" entry below), not something specific to this LoRA. Named for the record, doesn't change the verdict: base qwen3:4b carries the identical confound and this candidate still fabricated citations and never produced a report.

Cleanup same batch (per this project's disqualified-candidate hygiene rule): ollama rm deepdelve-qwen3-4b-combined-v2, deleted qwen3-4b-combined-v2-merged (7.6GB) and qwen3-4b-combined-v2.q8_0.gguf (4.0GB), ~11.6GB reclaimed. Kept the LoRA adapter itself (qwen3-4b-combined-v2-lora, 132MB) as the training artifact. config.yaml restored to deepdelve-gpt-oss:latest.

2026-08-19: v2 combined LoRA re-tested clean through api.backend: "ollama" — DISQUALIFIED, not confounded this time

Closes the "clean re-test, not yet done" caveat this entry carried since 2026-07-28. Re-merged the still-on-disk LoRA adapter (merge_and_unload() on CPU → convert_hf_to_gguf.py --outtype q8_0, 4.27GB) → redeployed as deepdelve-qwen3-4b-combined-v2 via llama.cpp-convert (already set up on /mnt/nuevovol/llm-models) → confirmed via direct curl against the live tag that think: true (NOT false) correctly isolates reasoning into its own field on the native /api/chat endpoint, think: false reproduces the identical known bug. Ran one clean trial via eval/evaluate.py --config pointed at a dedicated config (backend: ollama, enable_thinking: true, model deepdelve-qwen3-4b-combined-v2) against the standing visa/rent ablation benchmark.

One real process mistake along the way, worth recording: the first attempt at this re-test edited the LIVE ~/.deepdelve/config.yaml and launched eval/evaluate.py WITHOUT --config — evaluate.py never reads the live config at all, it always builds its own from src/tools/config_template.yaml unless --config is passed explicitly (_resolve_base_config_path, eval/evaluate.py:77). That entire ~47min run silently re-benchmarked deepdelve-gpt-oss again, not the candidate — a wasted run, caught only by checking the run's own eval_agent_config.yaml after the fact. Corrected by building a dedicated eval/qwen3-4b-combined-v2-config.yaml and passing it via --config, confirmed correct by re-checking the new run's own materialized config before trusting the result.

Result: score 0.25 (worse than the confounded run's 0.5), run finished on its own in 1513.8s (well under the 2820s ceiling, no resume needed). final_report.md is the deterministic-salvage banner — the model narrated in chat instead of ever calling write_workspace_file across its full writer-retry budget. findings.md repeatedly failed grounding on the same fabricated URLs (rentremote.com, nomadsembassy.com) across consecutive rebuild attempts, the same citation- fabrication shape as the original 2026-07-28 disqualification. Verdict confirmed, cleanly: the fine-tune's targeted objectives are real (held-out gains hold), but citation fabrication and writer-dispatch convergence — dimensions the combined reward never targeted — are still broken at 4B scale even with reasoning cleanly isolated from .content. Not a serving-layer artifact this time; a real capability ceiling for this base size on these specific failure modes.

Cleanup: ollama rm deepdelve-qwen3-4b-combined-v2, deleted the re-generated qwen3-4b-combined-v2-merged and qwen3-4b-combined-v2.q8_0.gguf, ~/.deepdelve/config.yaml restored to deepdelve-gpt-oss:latest/backend: openai/enable_thinking: false. LoRA adapter kept (training artifact). eval/qwen3-4b-combined-v2-config.yaml kept in the repo as a reusable template for the next candidate's clean re-test, since building it correctly (targeting the right line, not a comment that happens to contain the same substring) took one real mistake to get right too.

Root-caused same day: retries "not recovering" was NOT a model-capability problem — it was a structural bug. The known Qwen3 think-passthrough Ollama bug (see below) inflated every turn's token/char count 2-3x, blowing through context_budget_chars (50000, calibrated for nothink operation) in just 2 of the 8 configured completion-check attempts. tui.py's context-budget force-final path (run_state.attempt = 10**6) then jumped straight to run_completion_check's terminal branch, where _yield_to_starved_check's starvation window happened to be active and swapped the real, still-retriable problem (task_verification_flagged, citation fabrication) out for check_untracked_delegation — a check whose own docstring explicitly promises it "will NOT block this run from finishing." The forced-final path had no awareness of that contract, so a low-severity hygiene nudge got reported as the run's terminal blocker instead of the real correctness problem, which never actually got its intended 8-attempt budget to resolve.

Fixed (src/engine/completion.py, _yield_to_starved_check): added a never_final_blocker parameter, True for the check_untracked_delegation call site only (check_report_underuses_findings, a genuine correctness signal with no such guarantee, is unaffected — defaults False). Once ctx.attempt >= ctx.max_attempts (i.e. this cycle is already headed to the final branch regardless), the starved-check override is skipped so a documented-non-blocking check can never become the reported terminal reason. New regression case added to test_structural_checks.py's existing _starvation_guard_scenario; full suite re-run and passing.

Consequence for the DISQUALIFIED verdict above: it stands as recorded, but is now understood to be confounded — the model never got a real chance to resolve the citation-fabrication problem before being cut off. Re-benchmarking with this fix in place (and, ideally, once the Qwen3 think-passthrough bug itself is resolved or worked around) would be needed for a clean verdict; not done this session per explicit scope ("fix the bug, don't re-benchmark yet").

2026-07-26 (final entry, end of day): Ollama restored as the permanent serving backend, vLLM removed

Reverses the 2026-07-21 "Ollama dropped" decision, after the same-day vLLM re-test sweep above disqualified/discarded 9 candidates in a row, several with genuinely serving-layer-shaped symptoms (a silent zombie crash, intermittent empty-JSON tool calls, a garbled tool-call marker). Research (RESEARCH.md §11, primary-sourced against vLLM's own PR history and Feb 2026 blog, not just inferred from symptom shape) found: vLLM+bitsandbytes-on-ROCm support is real but young specifically on consumer RDNA GPUs (this hardware ran bitsandbytes exactly at its minimum required ROCm version floor); llama.cpp's HIP backend (what Ollama uses) turned up zero documented quantization-correctness issues across the same research, with years more real-world maturity on consumer AMD hardware.

Important, deliberately not oversimplified: most of today's actual DISQUALIFIED verdicts were NOT ROCm-caused (thin_coverage/narrate-instead-of-call reproduced identically on Ollama-hosted candidates too, including qwen3:8b on both backends) — this reverts the SERVING BACKEND to fix the flaky/crashy low-level tail, not a claim that it changes any model-capability verdict already reached. The two bugs that originally motivated dropping Ollama (Qwen3 think-mode passthrough, ollama/ollama#6155) are still real and unfixed — this is an explicit, informed tradeoff (accept those two known, narrowly-scoped bugs) in exchange for llama.cpp/HIP's overall serving-layer maturity edge on this specific hardware, not a claim that Ollama is now bug-free.

Cleanup: ~/.venvs/vllm deleted (~11GB), vLLM-specific HF cache checkpoints deleted (openai/gpt-oss-20b, ~13GB — DeepDelve's own actual dependencies, bge-reranker-v2-m3/ nli-deberta-v3-small/all-MiniLM-L6-v2, sharing the same HF_HOME cache location, deliberately preserved, NOT deleted). ~/.deepdelve/config.yaml restored to http://localhost:11434/v1/deepdelve-gpt-oss:latest, settings.skip_chat_template_kwargs reset to false. Full detail in the project_ollama_restored memory (supersedes project_ollama_dropped).

2026-07-26 (later still again): mistral-nemo:12b re-tested with the Mistral fix — DISQUALIFIED, same thin_coverage non-convergence pattern

Closes out the original 2026-07-21 BLOCKED verdict (see its own entry above, which already noted this exact fix as the remaining open re-test opportunity). Checked real weight size first, per the lesson from qwen2.5-coder/devstral: HF repo's mistralai/Mistral-Nemo-Instruct-2407 lists two packagings of the same ~24.5GB bf16 weights (consolidated single-file + 5-shard set); at the observed 4-bit ratio this comfortably fits (~8.3GiB actual, confirmed — matches the original pre-flight spike, which used this exact model). bitsandbytes 4-bit, mistral tool parser, settings.skip_chat_template_kwargs: true (today's fix, commit 9c8111f) — request reached the model cleanly this time, no 400.

Isolated tool-call smoke test: 6/8 clean across two batches (75%) — real structured arrays, no #6155-class bug — with two distinct intermittent failure shapes (a garbled tool-call marker prefix once, one full narration — "Cerebro has delegated the following tasks..." — with no real call once). Better isolated reliability than most other candidates today, not perfect.

Full DeepDelve benchmark run (the standing stress-test query): real, genuine engagement with the task — 13 sources fetched, 0/7 web-search failures, 8 findings recorded — clearly not a narrate-only or zero-engagement failure. But ended Retry budget exhausted (thin_coverage), final_report.md never written, all 4/4 completion-check attempts hitting the IDENTICAL thin_coverage problem — the same non-convergence signature already documented 3 times this project across different models AND backends (qwen3:4b on Ollama, qwen3:8b on both Ollama and vLLM): once thin_coverage fires, the Planner doesn't act on the corrective nudge and keeps repeating similar coverage instead of genuinely redelegating deeper. This is well-corroborated evidence even from one live run of THIS specific candidate, given how consistently this exact pattern recurs across unrelated model families and serving backends.

Verdict: mistral-nemo:12b DISQUALIFIED. The original 2026-07-21 BLOCKED status is now resolved — the chat_template_kwargs infrastructure issue is confirmed fixed (real research happened this time), but the underlying capability question comes back negative, joining the thin_coverage non-convergence family rather than escaping it. Not the same failure class as mistral:7b-instruct's own same-family verdict (not_delegated, narrate-instead-of-call) — this candidate genuinely engages with the task, it just can't escape thin-coverage retries once triggered.

Cleanup: vLLM server SIGTERM'd cleanly (zero orphan VRAM/processes confirmed), HF cache checkpoint deleted (~23GiB — this is a CLOSED, final verdict, unlike qwen2.5-coder's deliberately-preserved open item), ~/.deepdelve/config.yaml restored to the gpt-oss baseline.

2026-07-26 (later once more): devstral:24b discarded on hardware grounds without a pull attempt

Checked real weight size FIRST this time, per the lesson just learned from qwen2.5-coder:14b-instruct (don't spend time before confirming it fits). mistralai/Devstral-Small-2507's HF repo lists 94.3GB of .safetensors files, but that's misleading: it contains BOTH a single-file consolidated.safetensors (47.14GB, for mistral-inference) AND a separate 10-shard set totaling the same ~47.1GB (for HF transformers/vLLM) — the same weights packaged twice, not two different models. vLLM would only load the sharded set, so the real relevant footprint is ~47.1GB bf16.

Applying the empirically-observed bitsandbytes 4-bit compression ratio from qwen2.5-coder-14b earlier today (27.5GB bf16 → 9.9GB actual VRAM footprint, ≈0.36×): devstral's 47.1GB bf16 would land around ~17GB quantized — exceeding the entire 17.1GB card before any KV cache or activation overhead is even counted. Discarded on hardware grounds, same standard as qwen3.6's earlier discard — no pull attempted, no GPU time spent.

2026-07-26 (still later): qwen2.5-coder:14b-instruct — INCONCLUSIVE, wrong parser used initially, correct parser found but shows ~50% unreliable extraction

Qwen/Qwen2.5-Coder-14B-Instruct (confirmed real, not gated, via direct HF API check), bitsandbytes 4-bit. This candidate went through three distinct rounds before landing on an honest verdict — each correction below was caught by the user pushing back on an over-confident claim, not found independently, worth recording as-is.

Round 1 — hermes tool parser, crash, initially misdiagnosed. First launch at gpu_memory_utilization 0.7 failed cleanly with a graceful ValueError (needs 3.0 GiB KV cache, only 0.44 GiB available). Bumped to 0.9 (~14.3GiB budget) and retried — weight loading succeeded ("Model loading took 9.9 GiB"), then the process crashed silently with zero further log output (no "Available KV cache memory" line, which every other candidate today logged within ~2 seconds of the same point) — EngineCore found in zombie (Z) state, parent APIServer hung forever, no traceback anywhere. Initially attributed to a system-RAM OOM (the log's own "checkpoint size exceeds 90% of available RAM" warning looked suggestive) — user asked "are you sure that's the reason?", which prompted actually reading weight_utils.py's source: that warning is about an OPTIONAL read-ahead prefetch for network filesystems, correctly skipped here (local NTFS mount), its absence being the RAM-safe path, not a risky one. That theory didn't hold. Discarded the candidate on hardware-margin grounds instead (weights alone = 9.9GiB of ~15.9GiB usable budget).

Round 2 — user asked for a retry; the crash didn't reproduce. A clean second launch (same flags) got PAST the exact point that crashed before — "Available KV cache memory: 3.62 GiB", "Application startup complete". The crash was transient/intermittent, not deterministic — the "discard on hardware grounds" verdict from Round 1 was wrong on its own terms (it DOES start and run, just tightly). Ran the isolated delegate_tasks-shaped smoke test against the now-live server: 3/3 consistent failures, but a NEW, different shape than any other candidate today — the model produced a genuinely well-formed, correctly-structured tool call, just wrapped in <tools> tags instead of the <tool_call> tag the hermes parser requires.

Round 3 — wrong parser confirmed via primary sources, correct one found and installed. Web research + direct inspection of Qwen/Qwen2.5-Coder-14B-Instruct's own chat template confirmed: Qwen2.5-Coder (unlike vanilla Qwen2.5-Instruct) was never trained on the Hermes <tool_call> convention — it uses <tools> tags, and vLLM has no native parser for this variant (confirmed via direct grep of the installed tool_parsers/__init__.py registry). A real community project exists specifically for this gap: hanXen/vllm-qwen2.5-coder-tool-parser (12 stars, actively maintained, pushed 2026-04-29). Reviewed the actual parser source before installing it (392 lines, mirrors vLLM's own bundled parser structure, no network/subprocess/eval calls) — safe. Relaunched with --tool-parser-plugin/--tool-call-parser qwen2_5_coder/its accompanying --chat-template. Smoke test with the CORRECT parser: 2/4 consistent successes (real structured array), 2/4 returned arguments: "{}" (empty) despite finish_reason: "tool_calls" and a similar completion-token count to the successful runs — the plugin's own README independently documents that longer system prompts degrade this model's JSON-generation quality even under ideal conditions (49/50 vs 50/50 in their own "explicit" vs "minimal" mode tests), which is at least directionally consistent with an unreliable-under-load pattern, though the empty-{} extraction itself could be either the model or the third-party parser's own bug — not disambiguated, and not chased further given the time already spent on this one candidate.

Verdict: INCONCLUSIVE, not DISQUALIFIED and not a clean pass. Real capability under the correct format was never cleanly established either way — ~50% extraction reliability in isolated testing is too unreliable to call a pass, but it's also not the kind of confident, repeatable failure (not_delegated, fabricated content, #6155-class bug) other candidates today showed. A full DeepDelve benchmark run was never attempted (would need the custom chat-template/parser wired into ~/.deepdelve/config.yaml's vLLM endpoint, which only needs the server-side flags already proven above — no DeepDelve-side changes). Left as a genuine open item, not closed.

Cleanup: vLLM server SIGTERM'd cleanly (zero orphan VRAM/processes confirmed each round), HF cache checkpoint deleted (~28GiB) after the final round.

2026-07-26 (yet later): hermes3:8b DISQUALIFIED — narrates fake system messages instead of calling tools

Continuing the same day's vLLM re-test track. NousResearch/Hermes-3-Llama-3.1-8B, bitsandbytes 4-bit, hermes tool parser (confirmed correct, verified not gated via direct HF API check first). Isolated tool-call smoke test with the real delegate_tasks-shaped nested-array schema PASSED cleanly 3/3 (real structured tasks array, no #6155-class bug). No thinking mode in this model's chat template (checked directly, both default and tool_use template variants) — nothink is N/A for this candidate, same as llama3.2:3b/qwen2.5:3b-instruct.

The real DeepDelve benchmark run reveals a genuinely bizarre, distinctive failure: on its very first delegate_tasks attempt, the model narrated a FABRICATED system error as plain response text — "It looks like the prompt exceeded the maximum context length allowed by the model... I will try again with a shorter prompt." — then repeated this fake-retry narrative twice more, never once calling a real tool. Confirmed this error is entirely invented, not a real vLLM rejection: grepped the vLLM server's own log for the same time window — no context-length error of any kind exists there, and GPU KV cache usage was only 1-13% at that point, nowhere near the 16384-token ceiling. The model fabricated a plausible-sounding excuse out of nothing.

Confirmed via a second, independent run (a deliberately simple, different query, "what is the speed of light" — to rule out query complexity as a confound): same not_delegated root cause, different specific narration — this time the model wrote literal .delegate_tasks(tasks=[...]) pseudo-code as prose text, and after DeepDelve's own completion check explicitly told it "No delegate_tasks call was ever made," the model apologized in text ("Apologies for the repeated attempts. I understand the system's warning now.") and then immediately narrated the exact same fake call as text AGAIN instead of actually invoking the tool. Meets Model Evaluation Standard point 4 (more than one run): two independent runs, different queries, same not_delegated root cause each time, with two different specific hallucination shapes.

Verdict: hermes3:8b DISQUALIFIED. Same broad failure CLASS as mistral:7b-instruct's own same-day verdict (narrate instead of call, only surfacing under DeepDelve's real, much longer system prompt — the isolated short smoke-test prompt cannot catch this), but a more severe, specific manifestation: fabricating entirely fictional system/error text rather than just paraphrasing the intended call. Not a serving-layer issue — the isolated smoke test against the same endpoint/parser was clean.

Cleanup: vLLM server SIGTERM'd cleanly (zero orphans confirmed), HF cache checkpoint deleted (~15GB), ~/.deepdelve/config.yaml restored to the gpt-oss baseline.

2026-07-26 (even later): settings.skip_chat_template_kwargs fix unblocks Mistral-family vLLM candidates; mistral:7b-instruct DISQUALIFIED (narrate-instead-of-call, not the chat_template block)

Continuing the same day's vLLM re-test track. mistral:7b-instruct (mistralai/Mistral-7B-Instruct-v0.3, bitsandbytes 4-bit, mistral tool parser — same proven path as the original pre-flight spike) passed its isolated tool-call smoke test cleanly (3/3, real structured tasks array, no #6155-class bug — a genuine improvement over the original Ollama-era "rejected at schema stage" verdict). But the real DeepDelve benchmark run failed immediately with the SAME 400: "chat_template is not supported for Mistral tokenizers" error that already marked mistral-nemo:12b BLOCKED — and that entry had already predicted this would recur for mistral:7b-instruct and devstral:24b, now confirmed live for the first.

Root-caused directly at the vLLM source, not just re-confirmed from the earlier BLOCKED entry: vllm/tokenizers/mistral.py:161-162 — if request.chat_template is not None or request.chat_template_kwargs is not None: raise ValueError(...). Confirmed via vLLM's own PR history this is INTENTIONAL and permanent, not a version-fixable bug: PR #26358 (merged Oct 2025, well before this install's 0.25.1) explicitly changed this from a warning to a hard error, because Mistral's own mistral-common tokenizer library doesn't support overriding its built-in chat template in native Mistral tokenizer mode at all — checked both the model's own HF card (no vLLM serving guidance at all) and vLLM's docs/GitHub issues before concluding this; no server-side flag or --tokenizer-mode variant avoids it (mistral-nemo's own investigation already ruled out auto and hf modes).

Fixed on DeepDelve's side (commit 9c8111f): new settings.skip_chat_template_kwargs (default false), checked in _get_default_options() (src/engine/orchestrator.py) — when true, skips building extra_body (and therefore chat_template_kwargs/reasoning_effort) entirely. Explicit, user-set config, not model-family auto-detection (this project has repeatedly found string-matching heuristics unreliable — same philosophy as settings.specialist_model's own explicit design). Unblocks all three Mistral-family vLLM candidates at once (mistral-nemo:12b, mistral:7b-instruct, devstral:24b), not just this one.

With the fix applied, mistral:7b-instruct's real benchmark run got PAST the 400 wall — and hit a different, genuine capability failure: not_delegated, two independent runs, different queries, identical result. Run 1 (the standard stress-test query, "top 5 heuristic algorithms" + "Colombian holiday spending"): 4/4 completion-check attempts all not_delegated, zero findings, zero sources fetched, 145s. Run 2 (deliberately simple, "what is the speed of light," to rule out query complexity as a confound): identical not_delegated failure, zero sources, 77s. Root cause, confirmed from the real session log content: the model consistently NARRATES its planned delegate_tasks call as a literal markdown fenced code block in its response text (e.g. ```\ndelegate_tasks(tasks=[...])\n```) instead of actually emitting a structured tool call — even under --enable-auto-tool-choice. This reproduces the same "narrate instead of call/write" failure class already documented for Bonsai-8B and qwen2.5:3b-instruct (those at the FindingsWriter/writer-role stage; this one at the Planner's very first delegate_tasks, immediately) — DeepDelve's real, much longer system prompt evidently pushes this model toward narration in a way the short isolated smoke-test prompt didn't trigger. Meets Model Evaluation Standard point 4 (a discard claim needs more than one run): two independent runs, different queries, identical failure.

Verdict: mistral:7b-instruct DISQUALIFIED — genuine capability gap, confirmed NOT the chat_template_kwargs infrastructure block (that's fixed and confirmed working: the 400 is verifiably gone, the request reaches the model successfully both times). Distinct from mistral-nemo:12b's own verdict, which remains BLOCKED-not-yet-re-tested (the fix removes its blocker too, but it hasn't actually been re-run with the fix applied — that's a separate, still-open opportunity, not assumed to share this exact verdict).

Cleanup: vLLM server SIGTERM'd cleanly (rocm-smi --showpids confirmed zero orphans), HF cache checkpoint deleted (~14GB, unquantized bf16 cache — larger than the 4-bit runtime footprint since bitsandbytes quantizes on load, not on disk). ~/.deepdelve/config.yaml restored to the gpt-oss baseline (skip_chat_template_kwargs: false, openai_model: deepdelve-gpt-oss:latest).

2026-07-26 (later still): vLLM bake-off re-test resumed — qwen2.5:3b-instruct and llama3.2:3b both DISQUALIFIED, one result challenges the track's own core premise

Continuation of the same day's session, resuming the vLLM re-test plan (~/.claude/plans/moonlit-plotting-simon.md) after mistral-nemo:12b (BLOCKED), llama3-groq-tool-use:8b (DISQUALIFIED), and qwen3:8b (DISQUALIFIED) were already closed in earlier sessions. qwen3.6 (35b-a3b) was checked and discarded outright on hardware grounds before spending any GPU time — confirmed via a direct HF API blob-size query that its FP8 checkpoint alone is 37.5GB, more than double this hardware's entire 17.1GB VRAM budget, and no 4-bit pre-quant exists; even a hypothetical 4-bit re-quant (~18.7GB) still wouldn't leave real KV-cache headroom. Per Model Evaluation Standard point 6, discarded without attempting a pull.

• Qwen/Qwen2.5-3B-Instruct — DISQUALIFIED, new failure mode found on vLLM, not the original Ollama-era verdict's own reason. No quantization needed (bf16, ~6GB, easy fit). Tool parser: hermes (confirmed correct — qwen3_engine_tool_parser variants are Qwen3-only, don't apply to 2.5). Isolated single-arg tool call (web_search) parsed cleanly. A nested-array schema matching delegate_tasks' real shape (a tasks array of {task_name, instructions, agent_id} objects) consistently failed, 5+ independent reproductions: the model reliably emits genuinely malformed JSON for its own tool call — closes the "arguments" object but omits the FINAL closing brace for the outer {"name": ..., "arguments": {...}} wrapper (finish_reason: "stop", not "length" — not a token-budget cutoff, the model just stops one brace short). Verified this is a genuine model-generation defect, not a vLLM parser bug: tested hermes_tool_parser.py's extract_tool_calls directly in isolation against the real captured response text — the parser's own regex extraction is correct (confirmed via regex.findall on the raw content), it's the extracted JSON substring itself that's malformed at the source. Consistently reproduced (open brace count 4, close brace count 3) across independent live requests, not a one-off sampling fluke. This is a DIFFERENT failure mode than the original Ollama-based verdict (FindingsWriter never successfully calling write_workspace_file across 8 attempts) — same overall verdict (disqualified), new independent evidence via a completely different backend.

• meta-llama/Llama-3.2-3B-Instruct — DISQUALIFIED, and this result is significant beyond just this one candidate: it directly challenges the vLLM re-test track's own founding premise. Gated repo — required a real HF access-request-and-approval cycle mid-session (documented for process continuity: an hf auth login-verified valid token can still 403 on the actual /resolve/main/... download endpoint despite the account showing normal "Use this model" UI on the model page; the metadata API's visible file listing is not proof of real download rights — only the resolve endpoint is ground truth). Tool parser: llama3_json (confirmed via vllm/tool_parsers/__init__.py). Isolated tool-call smoke test with the exact same delegate_tasks-shaped nested-array schema used for the Qwen2.5 test above: tasks came back as a JSON-encoded STRING inside the arguments object, not a real array — the textbook #6155 shape (ollama/ollama#6155, "Support Nested Parameters for Tools") — reproduced 4/4 times, fully consistent. But this ran on vLLM, not Ollama. The entire premise motivating this multi-session vLLM re-test effort was that #6155 is an Ollama-serving-layer bug that may have wrongly disqualified mistral-nemo:12b/llama3-groq-tool-use:8b/llama3.2:3b on a backend artifact rather than genuine model incapability — this result directly disproves that hypothesis for llama3.2:3b specifically: the identical stringified-array shape reproduces independently on a completely different serving stack (vLLM's own llama3_json parser, not Ollama's tool-call handling at all). Confirmed via vLLM's own official documentation (docs.vllm.ai/en/latest/features/tool_calling.html, "Known issues" section for Llama Models): "The model can generate parameters in an incorrect format, such as generating an array serialized as string instead of an array." vLLM's own maintainers attribute this to the MODEL's generation behavior, not something their parser introduces or could fix — this is Llama 3.2's own limitation, independent of serving backend. mistral-nemo:12b and llama3-groq-tool-use:8b already have their own independent, non-#6155 disqualification reasons (real infra incompatibility and missing XML wrapper tags respectively), so this doesn't overturn either of those verdicts, but it does mean the #6155-is-Ollama's-fault framing that motivated re-testing all three should be treated as disproven for the one candidate where a clean head-to-head comparison was actually possible, not assumed to still be an open question.

Cleanup: both candidates' vLLM servers SIGTERM'd cleanly (rocm-smi --showpids confirmed zero orphan EngineCore processes both times), both HF cache checkpoints deleted immediately after each verdict (Qwen2.5-3B-Instruct ~5.8GB, Llama-3.2-3B-Instruct ~6.0GB) rather than batched to session end — per explicit user instruction this session to reclaim disk space right after disqualifying a candidate, not just at the end.

vLLM re-test track status after this session: mistral-nemo:12b BLOCKED, llama3-groq-tool-use:8b DISQUALIFIED, qwen3:8b DISQUALIFIED, qwen3.6 discarded on hardware grounds without testing, qwen2.5:3b-instruct DISQUALIFIED, llama3.2:3b DISQUALIFIED. Remaining untested: gpt-oss:20b (deliberately deprioritized to last per explicit user instruction — it's the already-trusted default, lowest information value to re-confirm), the schema-stage rejects (devstral, hermes3, qwen2.5-coder, mistral:7b-instruct), the writer-role failures (the other Gemma-4-12B, granite3.1-dense, phi4-mini), and Bonsai-8B/the GRPO fine-tune (lowest priority per the plan's own "attempt only if time remains" framing).

2026-07-26 (later): three more production bugs + yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF DISQUALIFIED

Continuation of the same day's session below. All committed and pushed to main.

Three more real bugs found via live testing, each caught by actually reading final_report.md after a smoke test, not just confirming a mechanism fired (see this session's own hardened standing rule: a completion check firing correctly is not the same as the report being correct):

• 5fab261 — real_grounding_problem/related checks in src/utils/grounding.py compared cited URLs against fetched URLs with plain string equality. A citation the model percent-encoded (%E2%80%93) didn't match a fetched URL stored with the raw Unicode en-dash, false-flagging a genuinely correct citation as hallucinated. Root cause of a real run ("explain the main theories for the extinction of the dinosaurs") coming back with an EMPTY report — "No extractable findings were identified" — despite 27 real sources fetched. Added _normalize_url (percent-decode + rstrip), applied at every URL-comparison site in the module. Regression test pinned against the exact real citation/URL pair from the incident; confirmed failing on pre-fix code via git stash, passing post-fix.
• 6edd9ee — delegate_tasks is one shared closure/quota across every tier (Planner's top-level dispatch and every Tier-2 specialist's own dispatch to its Analyzer children draw from the same global tool_quotas_ctx counter, no per-task sub-budget anywhere). A live smoke test on a trivial single-fact query ("what is the Cretaceous–Paleogene boundary") saw ONE WebSearcher task call delegate_tasks 6+ times in a row, despite its own prompt saying "ONE authoritative source is sufficient" and "STOP EARLY" — consuming most of the run's entire global budget on one facet. Added settings.specialist_delegation_cap (default 3), enforced via delegation_depth_ctx (> 0 == a Tier-2 specialist, never the Planner) in src/engine/orchestrator.py. A real bug was caught in the fix itself before committing, via the same live-testing discipline: the first version used ctx.set(new_int) to update the counter across separate delegate_tasks calls, which silently never accumulated (8 Analyzer children spawned from one task, zero rejections) — the SDK dispatches each tool call as a separate asyncio.Task, so a .set() inside one call only mutates that Task's own context copy. Fixed by mirroring task_fetched_urls_ctx's existing mutable-list pattern (set once per dispatch, mutated in place). A second live run confirmed the cap actually holds.
• d88e445 — check_task_verification_flagged (VERIMAP Phase 1, shipped earlier the same day) had no notion that a flagged task could be superseded when the Planner renames it on retry instead of reusing the task_name. A live run got a flagged task renamed 3x ("...definition" → "...definition (narrow)" → "...definition (peer-reviewed)" → "...definition (peer-reviewed source)"); two renamed variants actually succeeded (verified, real Wikipedia-backed findings), but the check kept re-nudging the ORIGINAL stale names specifically — burned the entire completion-check retry budget (8/8), zero report ever written, worse than the bug this session started from. Fixed: _update_task_verification now does a second pass, downgrading a flagged entry to a new "superseded" status when its dispatched instructions (from run_state.data["dispatched_tasks"]) closely match an already-verified task's instructions, reusing the existing _looks_like_renamed_task helper. Also reworded the check's own directive to say "reuse the EXACT task_name" (defense in depth, matches the wording already applied to two other checks earlier the same day — soft instructions alone don't reliably hold on this hardware, which is why the structural supersede detection is the primary fix). Live-verification status: PARTIAL. A third live run of the same query showed NO repeat of the stale-nudge loop (real improvement), but the run was still going after 35+ minutes (Planner-level over-fanning into multiple top-level research angles — a separate, unrelated, still-open problem neither this fix nor the delegation cap constrains) and was killed before reaching a final report. Mechanism-confirmed (no repeat loop), not full-outcome-confirmed (never saw a final_report.md from this exact fix). Re-verify with a full run if this failure mode recurs.
• e60e334 — found while doing Model Evaluation Standard point 1 (confirm operating mode via a raw API test) on the Gemma candidate below, but it's a general engine fix: _get_default_options (src/engine/orchestrator.py) now also sends reasoning_effort: "none" alongside the existing chat_template_kwargs.enable_thinking whenever thinking is disabled — some models (see verdict below) ignore the latter entirely over Ollama's OpenAI-compat endpoint but respect the former. Confirmed harmless against the gpt-oss baseline (unchanged short harmony-format reasoning note).

New, separate, not-yet-acted-on finding: _stub_reason did NOT flag a ScienceDirect fetch that came back as almost entirely Adobe-Analytics tracking-script JSON, not real article content — no stub key at all in fetched_urls. Unlike a google.com fabricated-citation fetch in the SAME run, which WAS correctly caught as a stub. A real gap in _stub_reason's pattern coverage for this specific bot-wall shape; low priority, didn't actually poison a citation this run.

yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF — DISQUALIFIED, run twice, identical failure signature both times. Not the same candidate as the existing "Gemma 4 12B" row below (that one is SetneufPT/Gemma4-12B-IT-QAT_Q4_64K_16GB-GPU, a different publisher/quant of the same base architecture) — this is a separate fine-tune, evaluated separately.

• Base: google/gemma-4-12B-it, 11.9B params. Pulled as hf.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF:Q4_K_M (7.4GB). Derived tag deepdelve-gemma4-12b:latest created with PARAMETER num_ctx 16384 (the raw pulled tag has no context override — Ollama would otherwise silently serve it at its small default, invalidating any test per this project's own context-floor rule). OLLAMA_NUM_PARALLEL=1 already set system-wide, applies automatically.
• Model Evaluation Standard point 1, confirmed via direct curl against the real /v1/chat/completions endpoint before any benchmark: plain completion clean; tool-calling clean (correctly-shaped tool_calls, empty content). chat_template_kwargs.enable_thinking: false does NOT suppress thinking — the reasoning field stayed populated regardless (same passthrough-bug class already documented for Qwen3). Root-caused to a working alternative: reasoning_effort: "none" on the SAME endpoint DOES cleanly suppress it — this is what motivated e60e334 above. Confirmed clean (no reasoning field, correct tool_calls) with the fix applied, before running any real benchmark.
• Two live runs against DeepDelve's actual engine, different queries, identical failure: run 1, the same "Cretaceous–Paleogene boundary" query used to verify the delegation-cap/supersede fixes above (1321s, Report: NOT WRITTEN); run 2, a deliberately narrow single-fact query ("what is the speed of light," 934s, Report: NOT WRITTEN) chosen specifically to rule out query complexity as a confound. Both runs ended identically: "Task forcefully aborted: Agent trapped in loop. Quota exceeded multiple times for write_todos."
• Root cause, confirmed from the real session log's actual tool-call arguments (not guessed): the model calls write_todos with BYTE-FOR-BYTE IDENTICAL arguments repeatedly (the same fully-[x]-checked todo list, verbatim, call after call) — not incremental revision, a genuine repetition loop. It continues even AFTER the tool starts returning "Error: Quota reached... You MUST summarize what you've done and state clearly that you had to stop due to quota limits" — the model ignores this explicit corrective instruction and calls write_todos again with the identical content. All research work was actually done (todos fully checked, real sources fetched — 17 and 10 respectively, 0 web-search failures both runs) but the model never transitions to writing the final report; DeepDelve's own anti-loop safety net (quota + forced abort) is what stopped the run, not the model recognizing completion.
• Meets the Model Evaluation Standard's point 4 (a discard claim needs more than one run): two independent runs, different queries (one deliberately simple to rule out complexity as a confound), identical failure signature.
• Distinct failure mode from every other disqualified candidate documented so far — not the thin_coverage Planner non-convergence pattern (qwen3:4b/qwen3:8b), not citation fabrication (InternScience/Agents-A1-4B, the fine-tuned qwen3:4b), not a writer-role tool-call failure (Bonsai-8B, qwen2.5:3b-instruct). This is a genuine inability to break out of a literal identical-tool-call repetition loop even when directly instructed via a tool-error message to stop — the most severe agentic-reliability failure documented in this project's bake-off so far.
• Not a serving-layer bug — nothing here resembles the Qwen3/mistral-nemo-class Ollama passthrough bugs (point 3 of the standard): nothink and tool-calling were both independently confirmed clean at the raw API level beforehand, and the failure only appears once the model is actually driving the full multi-turn agentic loop.
• Closed out: ~/.deepdelve/config.yaml restored to the gpt-oss baseline (openai_model: deepdelve-gpt-oss:latest). Both Ollama tags for this candidate deleted (deepdelve-gemma4-12b:latest and the raw hf.co/yuxinlu1/...:Q4_K_M pull it was derived from, ~7.4GB reclaimed — they shared blobs, not 2×7.4GB), same cleanup convention as every other disqualified candidate's checkpoint (e.g. qwen3:8b's vLLM checkpoint deletion above). No trace of this candidate left on disk or in ollama list.

2026-07-26: Planner redelegation-loop fix + two new production bugs found by live testing

Addressed 3 of the "still open" items from the 2026-07-24 session (excluded: the deferred Ternary-Bonsai-27B/Vulkan fork work). All committed on main:

• 999a254 — Root-caused why the Planner renames the same research angle across retries instead of redispatching under the same task_name (e.g. background_heuristic_algorithms → _refined → _final; recurred as 43 calls for one angle under --resume-run). Cause: check_thin_coverage/check_uneven_task_investment's own directives said "redelegate ... phrased differently," which a model conflates with inventing a new task_name — nothing anywhere said otherwise. Fixed the directive wording, and added a non-blocking difflib-based similarity nudge in delegate_tasks (_looks_like_renamed_task, src/engine/orchestrator.py) that flags a likely rename against the run's own prior dispatches. Live-confirmed firing 9/20 then 4/6 real delegate_tasks calls across two separate live runs — reduces but doesn't eliminate renaming (a prompt nudge alone can't guarantee compliance). Also live-confirmed (same runs) that check_untracked_delegation's once-per-run cap (shipped 2026-07-24) holds under real pressure, and that check_report_underuses_findings fires live for the first time ever (previously unit-tested only).
• 1ed5de7 — A live smoke test hit a more severe case of the empty-response problem than the 2026-07-24 fix assumed: FindingsWriter produced nothing usable on BOTH the original dispatch AND its immediate retry, six consecutive completion-check attempts in one run, exhausting the budget with findings.md never written despite 61 real findings existing the whole time. Added a deterministic (non-LLM) salvage path: _dispatch_writer_review_fix now accepts a deterministic_fallback, wired only from the FindingsWriter call site as _build_findings_source_material's own real evidence text — written directly as findings.md when both attempts fail, instead of losing the cycle. Found and fixed a second, unrelated bug while verifying this against the real failing run's actual data (not synthetic tests): extract_cited_urls (src/utils/grounding.py) didn't stop at a trailing backtick, so the common `URL` inline-code citation style broke every fetched-URL match — would have defeated the new fallback on the exact content it exists to rescue.
• 554cfc2 — Caught by actually reading a live run's final_report.md content (not just confirming completion checks fired mechanically, per the standing project practice): a balanced 2-facet query (green tea + Roman Empire, both genuinely "covered" per RunState.coverage()) produced a final_report.md that was 100% about one facet — the other vanished entirely at the FindingsWriter consolidation stage, and NOTHING caught it (check_thin_coverage saw balanced research volume; check_report_underuses_findings compares the report against findings.md, which had already lost the data). New check_findings_underuses_evidence (src/engine/completion.py, right after check_stale_findings) compares findings.md's cited URLs against RunState's real per-task research record and fires when a covered task has ZERO of its real URLs represented. Replayed directly against the real failing run's data to confirm it would have caught the omission. Not yet observed firing organically on a fresh live run.
• 34cb72c — Added a 6th GRPO reward-function dimension, findings_underuses_evidence_response_reward (finetune/reward.py), mirroring the check above, for if/when fine-tuning resumes. Inert prep only (self-tested, not wired into train_combined_grpo.py's training loop, no training launched) — that would need per-task URL grouping threaded through the synthetic training-data generation pipeline too, a separate, bigger change.

Standing lesson reinforced this session: confirming a completion-check fired via _run_state.json proves the mechanism ran, not that the report is actually good — read the real artifact content too before calling a run's outcome confirmed.

Later the same day — a fresh live smoke test (2008 financial crisis + mRNA vaccines, --depth standard) confirmed the deterministic FindingsWriter salvage (1ed5de7 above) firing organically for the first time, live, outside any replay/unit test. But the run still failed overall (findings_ungrounded, no report written) for a separate, real reason: FindingsWriter repeatedly cited a fabricated near-duplicate insidetx.com URL across 7 independent dispatches, despite the grounding pipeline already embedding a [SYSTEM VERIFICATION WARNING: ...] directly into that finding's own summary text at generation time — the exact same shape already logged 2026-07-24 with a calendarr.com URL, now confirmed to recur with a different domain. Researched properly before fixing (RESEARCH.md §10): embedded negative-instruction warnings are documented as fragile by mechanism (naming forbidden content can prime its reproduction — the "ironic rebound" effect, arXiv:2511.12381; negation-following degrades in small models, arXiv:2601.21433), and CRAG/Self-RAG (primary papers, arXiv:2401.15884/arXiv:2310.11511) both structurally filter flagged evidence out before the generator sees it rather than annotate-and-hope.

• _is_citable_finding (src/engine/completion.py) — the shared predicate feeding _build_findings_source_material, _uncited_task_names, and _find_propagated_bad_content — now excludes findings carrying a [SYSTEM VERIFICATION WARNING marker, exactly mirroring its existing [SYSTEM RELEVANCE WARNING exclusion (2026-07-21). This deliberately REVERSES a 2026-07-22 design decision that explicitly chose NOT to exclude verification-flagged findings, reasoning they "may still coexist with other real, usable content" — a bet the live recurrence above shows doesn't hold. Since run_completion_check computes _build_findings_source_material(run_state) exactly once and reuses the identical string as both the FindingsWriter dispatch instructions AND the deterministic-salvage fallback (1ed5de7 above), this one change fixes both the recurring-fabrication problem AND the "salvage inherits poisoned evidence" problem simultaneously — no separate patch needed for either path. _should_cache_finding (src/engine/orchestrator.py, the RAG-cache write gate) already treated this warning as disqualifying, so _is_citable_finding was the odd one out relative to existing project precedent, not inventing a new policy. Verdict-matrix-adjacent unit test flipped (test_structural_checks.py) and replayed directly against the real 2026-07-26 failing run's actual _run_state.json: confirmed the fabricated insidetx.com citations no longer appear in the reassembled evidence post-fix. Tests pass, ruff clean. Explicitly did NOT build a cross-attempt confirmed-bad-URL blocklist (CiteGuard-style, RESEARCH.md §10's secondary recommendation) — the two live incidents fabricate different URLs each retry, not byte-identical repeats, so the exclusion fix above already covers both flagged problems without the added persisted-state/resume-carryover machinery a blocklist would need; kept as the fallback if this fix proves insufficient on a future run.
• Two further research-flagged ideas (VeriCite-style NLI-based evidence filtering at assembly-time, VERIMAP-style per-subtask verification functions) were deliberately NOT built this session — both real, well-scoped future directions, written up in Pending below rather than folded into this fix's diff.

Live-confirmed same day, on a fresh run (CRISPR-Cas9 mechanisms + 1929 Wall Street Crash, --depth standard): attempt 1 flagged a fabricated blinkist.com citation via findings_ungrounded — and, unlike the pre-fix insidetx.com run (a 7-attempt loop on the exact same bug), it never recurred in either findings.md or final_report.md (confirmed by direct grep: zero occurrences in both). FindingsWriter converged normally on the next attempt instead of looping. The run's eventual outcome (report_underuses_findings, then max_run_minutes mid-retry) was a separate, pre-existing, correctly-handled failure mode — the check fired accurately, the run just ran out of wall-clock budget incorporating more sources — not a regression or a new bug.

Later the same day — went through the two research-flagged Pending ideas above. The VERIMAP-style per-subtask verification-function reframing stays deferred as a real future architectural direction (unchanged, still in Pending). The NLI-based evidence filtering item turned out to be based on an incorrect assumption, caught by re-reading the actual dispatch code instead of trusting the earlier note: _run_single_task (src/engine/orchestrator.py) already runs real_grounding_problem — the FULL grounding pipeline, including NLI contradiction detection via nli_unsupported_problem — for Tier-2 Searcher dispatches. The real, narrower gap was that Tier-3 Analyzer-leaf dispatches (DocumentAnalyzer/DataAnalyzer) never got this at all — only a narrow reconstructed-URL check added 2026-07-19 for one specific bug, with no stub detection, no quote-fidelity, no content-level check, no NLI. Analyzer findings (reading and summarizing a single fetched document — exactly the shape most likely to misreport a figure or misattribute a stat) had materially weaker grounding coverage than Searcher findings, for no principled reason.

Fixed: added the same real_grounding_problem(final_text) call already proven via the Searcher-tier check to the Analyzer-leaf branch too (src/engine/orchestrator.py), alongside (not replacing) the existing reconstructed-URL check — the two catch genuinely different things (reconstructed-URL catches "cited a different-but-real URL, not the one you were given"; real_grounding_problem catches "URL matches, but the claim doesn't say what the source says"). Both append to the same verification_warnings accumulator using the identical [SYSTEM VERIFICATION WARNING marker, which _is_citable_finding already excludes structurally (this session's earlier fix) — no changes needed to the exclusion mechanism itself. Zero new machinery: real_grounding_problem/nli_unsupported_problem/the lazy CPU-only NLI singleton all already existed and were already proven in production via the Searcher-tier check. Ladder rung 2 (reuse, don't rebuild) — the original Pending note's "needs new claim/source-window extraction" premise was simply wrong. Tests pass (no new unit test added, consistent with the reconstructed-URL check's own precedent of relying on live verification for dispatch-loop wiring plus real_grounding_problem's own already-thorough existing test coverage for the logic itself), ruff clean.

Live-confirmed same day on a fresh run (Pfizer/Moderna vaccine efficacy statistics + Irish Potato Famine causes, --depth standard): two Analyzer-tier findings got flagged with problem types (claim_unsupported, stub_source) that were structurally impossible before this change — the old Analyzer-leaf check could only ever produce a reconstructed-URL message. Confirms the new real_grounding_problem call is genuinely executing for Analyzer dispatches now, not just present in the diff. nli_unsupported specifically did not fire this run (an earlier pipeline stage caught the problem first, since real_grounding_problem short-circuits on the first hit) — the wiring is confirmed correct, NLI itself being the deciding factor is still an open, narrower confirmation gap.

VERIMAP Phase 1 — a structural per-task verification ledger, same day, after scoping in a dedicated plan. RESEARCH.md §9 flagged VERIMAP (EACL 2026, arXiv:2510.17109) — a planner encodes an explicit verification function per subtask, executed by a separate Verifier before a Coordinator proceeds. Confirmed via exploration that DeepDelve's 23-check pipeline produces exactly ONE Verdict per attempt for the WHOLE run — no per-task pass/fail record existed anywhere, and several bugs fixed this session (the starvation guard, check_uneven_task_investment, check_findings_underuses_evidence) are all individually-patched symptoms of that same missing dimension. Design tension identified and resolved before building anything: VERIMAP's own mechanism has the PLANNER author each verification function — directly conflicting with RunState.coverage()'s own documented principle that DeepDelve avoids handing local models new structured conventions to follow, precisely because they've repeatedly proven unreliable at it. Reconciled by having the ENGINE compute the ledger structurally from already-existing ground truth (_is_citable_finding) instead — keeping VERIMAP's real contribution (a task has its own checkable verification state) while dropping the part of its mechanism that doesn't fit DeepDelve's own hard-won lesson.

• New run_state.data["task_verification"] ledger, keyed by task_name ({"status": "verified"|"flagged", "reason", "checked_at"}), recomputed fresh every completion- check attempt by _update_task_verification (src/engine/completion.py) from findings + _is_citable_finding — purely additive, no new Planner-facing field, no new prompt convention. A task with no findings yet is left out of the ledger entirely (still pending, not a problem).
• New check_task_verification_flagged (COMPLETION_CHECKS, right after check_thin_coverage) — the first genuinely task-scoped check in the pipeline: fires when a task's EVERY finding was excluded by _is_citable_finding, naming that SPECIFIC task in its directive rather than nudging the whole run generically. Distinct from check_thin_coverage (zero findings at all) and check_uneven_task_investment (uneven counts across covered tasks) — this catches "produced findings, but every one turned out fabricated/off-topic/contradicted." settings.task_verification_check (config_template.yaml), default enabled.
• ARCHITECTURE.md's resume-carryover allowlist updated in both src/engine/tui.py locations (_resume_run and run_cli), per its own checklist for a new persisted RunState.data key.
• Explicitly deferred (Phase 2, see Pending): actually independent per-task redispatch that bypasses the Planner's own turn — this session's ledger still only produces one whole-run Verdict per attempt, same shape as every other check; the deeper dispatch-loop rework is real, separate scoping work, worth doing only after seeing how often "flagged" recurs on real runs.

Tests: new _update_task_verification/check_task_verification_flagged scenario, new verdict- matrix row, resume-carryover extended for both TUI and run_cli copies (mirroring findings_written_citable_count's own precedent exactly). Tests pass, ruff clean.