Roadmap

DeepDelve Roadmap

Status as of 2026-08-20.

This file is organized into: a standing methodology section (applies to all future model verdicts), then History (investigation narrative — informational, not a task list), Completed (shipped fixes/features), Pending (not started), Rejected (evaluated and discarded), and Stretch (deferred/optional work, currently all fine-tuning).

Model Evaluation Standard

Written after the user pushed back on two real fairness gaps found by re-reading the bake-off log critically rather than taking past "discard" verdicts on trust: (1) the heterogeneous-tiering entry above measured a foreseeable VRAM-thrashing result instead of catching it at design time, and (2) MiniCPM5-1B's own FINAL VERDICT run (below) swapped the Planner/Builder off gpt-oss:20b onto mistral-nemo:latest to free VRAM — meaning that verdict wasn't actually isolating the specialist model as the one variable under test; some of what got blamed on MiniCPM5-1B (the uncorrected "[Authors' names]" placeholders, specifically) was explicitly attributed to the swapped-in Builder failing to catch it, not to MiniCPM5-1B itself. Neither gap was hidden — both are documented in the entries themselves — but neither was caught BEFORE being treated as a concluded verdict, which is the actual complaint. Going forward, a candidate is not "discarded" or "adopted" until it clears all of the below:

1. Confirm the operating mode actually reaches the model before scoring anything. Don't infer a feature (nothink mode, tool-calling format, context length) from a model card or vendor docs alone — prove it with a raw API-level request (a direct curl/SDK call showing the expected field, e.g. enable_thinking:false producing zero <think> content) BEFORE running any full DeepDelve benchmark through it. This is exactly what MiniCPM5-1B's entire think-mode saga should have started with, and what caught the Qwen3-family Ollama passthrough bug only after several models had already been scored under it.

2. Isolate the candidate as the only variable. Every other role (Planner/Builder/ FindingsWriter/PeerReviewer) stays on the project's known-good baseline (gpt-oss:20b) unless the candidate itself IS one of those roles. If VRAM genuinely forces a swap elsewhere in the pipeline for a test to run at all, that test cannot produce a clean verdict on the candidate — it can only be reported as informational, and the entry must say so explicitly. MiniCPM5-1B's FINAL VERDICT run above did not meet this bar — the Planner/Builder was swapped to mistral-nemo for VRAM. In principle that calls for an isolated retest; in this specific case the user has explicitly decided NOT to pursue that retest (see the MiniCPM5-1B entry's "Retest explicitly NOT queued" note and the "Heterogeneous role tiering" closure note below) — pairing gpt-oss:20b with any small specialist model is a closed strategy on this hardware regardless of which small model fills the slot. The general rule (isolate before verdicting) still applies to any FUTURE candidate; it does not retroactively reopen MiniCPM5-1B.

3. State the serving backend and version alongside every verdict. "Disqualified" must mean the MODEL failed, not that Ollama's serving layer mishandled it — the nested-array stringification bug (ollama/ollama#6155, affecting mistral-nemo/llama3-groq-tool-use/llama3.2:3b) and the think-mode passthrough bug (Qwen3 family) both mean some existing README/ROADMAP disqualifications may need a backend-corrected retest before they're trustworthy, not just the ones already flagged for the planned vLLM re-run.

4. More than one run before a verdict, when the result is a discard. A single run's failure can be a real capability ceiling or an unlucky decode/retry cascade — this project's own log has both (qwen3:4b's multiple redispatch attempts vs. a genuine hard ceiling). A clean pass can still be reported off one run; a discard claim should be corroborated by at least a second run before being written up as final, or explicitly marked "single-run, not yet corroborated" if time didn't allow a second one.

5. Keep a verdict changelog instead of silently overwriting. If a verdict was reached under a later-found-flawed methodology (wrong operating mode, confounded pipeline, backend bug), don't delete or rewrite the old entry — mark it superseded and link to the corrected retest, so a reader can see which methodology produced which conclusion. This is why MiniCPM5-1B's entry already has separate "think-mode" and "FINAL VERDICT (nothink)" sub-entries rather than one overwritten verdict — keep doing that, and extend it to the confound flagged in point 2.

6. A candidate that can't fit the project's ~16K-token context floor is discarded outright on hardware grounds, not proportionally rescaled to squeeze it in — user decision, 2026-07-21. config_template.yaml's context_budget_chars: 50000 is explicitly calibrated as "safe margin under a 16K-token num_ctx" (see README.md's Context management section and the get_context_budget() docstring, src/engine/orchestrator.py) — this is the project's assumed minimum operating context, not a soft target. When MiniCPM3-4B's real per-token KV cost on this hardware capped its feasible serving context at 6144 tokens (well under that floor), the response was to proportionally scale context_budget_chars down to fit — this is now the wrong call going forward. Doing so tests the candidate under a context regime the project doesn't actually run at, and produces one of two uninformative outcomes: a pass that doesn't generalize to any real DeepDelve usage, or a failure that's actually a context-fit problem miscounted as a capability problem. Going forward: check the candidate's actual max feasible serving context on this hardware (via vLLM's own KV-cache-budget error message, same as this evaluation did) BEFORE running any benchmark; if it can't clear ~16K tokens, discard immediately with the reason recorded as "insufficient context on current hardware," and revisit only if better GPU/VRAM becomes available — don't rescale the project's own safety margins to accommodate it. Clarified 2026-07-21, llama3-groq-tool-use:8b: this point targets a HARDWARE-forced squeeze (a candidate whose architecture could serve more context but this GPU's VRAM/KV-cache budget won't allow it) — it does NOT apply to a model whose own native max_position_embeddings is simply small by training (llama3-groq-tool-use:8b's is 8192, a real fixed fact about the model, not something any amount of better GPU/VRAM would ever change). The user's own distinction: a permanent model-level limit is worth actually testing at its real native ceiling — only the hardware-driven, potentially-temporary kind gets the outright-discard treatment. Test the candidate at its true native context in this case, don't discard on point 6 grounds.

7. A dense model above ~12B parameters is discarded on raw-throughput hardware grounds alone, before any tool-calling test, unless it uses an efficient quantization scheme (MXFP4 or similar) — added 2026-08-29, after two independent architectures confirmed the same ceiling. gemma4:12b (11.9B dense) was already disqualified at 32.2 tok/s on this RX 9060 XT. The same day, devstral:24b (23.6B dense, Llama-based) measured 18.4-19.0 tok/s and mistral-small3.2:24b (24.0B dense, Mistral3 arch) measured 19.3 tok/s — two unrelated architectures landing within 1 tok/s of each other, both well below gemma4:12b's own already-disqualifying ceiling at roughly double the parameter count. This is a genuine hardware compute ceiling for dense transformers at this size class on this GPU tier, not a per-model quirk — gpt-oss:20b's own 62.7 tok/s and nemotron-3-nano's 86.2 tok/s only clear it because both use an efficient scheme (MXFP4 quantization, a hybrid Mamba/attention architecture respectively), not because of raw parameter count. Before pulling any dense model above ~12B for this hardware, check for an efficient quantization/architecture first — a plain Q4_K_M (or similar) dense quant at this size class can be discarded on a single raw throughput measurement, the same "check before benchmarking" discipline point 6 already applies to context. A local MoE's ACTIVE parameter count does NOT change this VRAM/throughput calculus either — every expert's weights must stay resident in Ollama regardless of how many are used per token, so a "30B total / 3B active" MoE costs VRAM like the full 30B, not the 3B (confirmed against Nemotron-3-Nano-30B-A3B, GLM-4.5-Air, Qwen3-30B-A3B, all discarded on this exact basis the same day) — there is no cheap-active-param shortcut to a bigger effective model on this hardware; gpt-oss:20b's efficient MXFP4 quantization of its whole ~21B is what makes its footprint small, not its architecture shape.

History

Moved to the wiki, 2026-08-20, to keep this file focused on what is currently open:

→ Changelog

Completed

Finished items moved to the wiki, 2026-08-20, to keep this file focused on what is still open:

→ Completed

Pending

Genuinely open items only. Closed items that were still narrating their full resolution history here got moved out, most already live in the wiki's Completed or Changelog; anything not yet migrated is tracked in session_status/CURRENT.md until the next wiki pass picks it up.

• Claim-support judge for standard-style [N] reports (opened 2026-10-01). The default [N] + numbered-Sources style had NO claim-level check: a fabricated claim cited [11] passed claim_unsupported (fixed, 2a1d951: segment checks now see [N]). NLI and topical relevance still skip [N] segments on purpose: the deberta-v3-small NLI called a correct claim a 0.93 contradiction and is at chance on a 61-pair labelled set (AUROC 0.62-0.66; eval/claim_support/). MiniCheck (deberta-v3-large, already installed, opt-in editorializing_problem) scores AUROC 1.00 on wrong-source negatives but 0.65 on hand-labelled bad pairs and is too strict on multi-fact claims (a cutoff of 0.1 flags 4/23 correct claims). 2026-10-01 follow-up: MiniCheck paper (arXiv 2404.10774, 30 pp, read in full incl. appendices) reports claim decomposition gives no consistent gain (Table 5: MiniCheck-FT5 -1.4 BAcc) at 2-4x cost, so the atomic split is NOT built; its multi-doc rule is max over docs, threshold fixed at 0.5, chunks ~400 tokens. Claim normalization (row-label prefix, '7 %', '€400 000', spaced punctuation; eval/claim_support/score.py norm) moved AUROC on bad pairs only 0.65 -> 0.66 and false flags at 0.1 from 4/23 to 3/23: not the cause. Still no defensible threshold (n=8 bad pairs). Open: more labelled reports + second labeller, per-source max for grouped citations ([7][8][9] are checked against the union today), more labelled reports and a second labeller before any threshold goes live (n=2 reports, single labeler so far). Currency swaps (EUR vs USD) are covered deterministically by _currency_swapped. Papers read in full for this (not yet in the wiki Literature Review; none validates a claim-support threshold): arXiv 2605.06635, 2605.27700 (reference existence only), 2602.13855 (position paper, no experiments), 2609.35774 (completeness, not support), 2510.17853 (citation retrieval), 2602.15871 (reference existence only). 2026-10-02 update (n=109 pairs, 57 supported / 22 natural-bad / 30 cross; eval/claim_support/{extract_new,label_new,score,report}.py): added 3 reports (Raft/Paxos 224216, AI Act 220924 + 214601), 48 new pairs hand-labelled by one labeller (Claude; each checked by grepping the cited source; 10 ambiguous generalising claims skipped). MiniCheck (CPU, no normalisation) AUROC vs natural-bad 0.81 (was 0.65 at n=8), vs cross 0.99. Cutoff 0.1 catches 15/22 bad but flags 8/57 supported (14%); 0.5 catches 19/22, flags 22/57 (39%). Still too noisy to block a run; at most advisory. New bad pairs are mostly "true elsewhere, absent from the cited source" (EU fine tiers, an AI Office date, a Paxos Commit detail), which is the case the check exists for. Open: a second, independent labeller; per-source max is implicit (pairs are per [N]) but the live check has no grouped-citation scoring yet.

• Dated-status vs own-timeline check: prototyped offline, NOT built (2026-10-02). Target: baseline_1's "high-risk fully applicable 2 Aug 2026" vs its own "stand-alone high-risk 2 Dec 2027". Sentence pairs with different single dates and content-word Jaccard >= 0.3, run over every report in research_output/: it fires on every calendar/version table (Colombia holidays 153 pairs, Rust release dates, Iceland population) and on legitimate staged deadlines (Annex III 2027 vs Annex I 2028, Jaccard 0.43-0.71, higher than the real contradiction's 0.38). No deterministic discriminator separates them with one positive example; any rule tuned to it overfits. Would need an LLM/NLI pairwise judge plus a labelled set of self-contradicting reports first.

• Verification of the 2026-10-01 low-effort A/B (done): the writer now converges (ec688b7), both runs ended "unverified" on two checker false positives (spaced thousands EUR 400 000, resolved [N] not counted as a citation), fixed in 3ccd679 and re-verified offline only. Still open: a live rerun to confirm a clean verified run, the baseline arm of the A/B on current code, the settings.thinking_effort decision for orchestrator.py:766, and the cross_source_contradiction "5 %" vs "3%" flag.

• Pipeline planner facet cap (d5d338b): when entities x measures exceed 6 facets the planner prompt now asks for one facet per entity covering every measure (h25 dropped fertility rate for all three countries). Prompt-only, replayed 5x on h25 (4/5), NOT re-run end to end on h19-h27.

• Stage-graph pipeline prototype (branch stage-graph-prototype, src/pipeline/) — evaluate, then decide how it merges with the agent loop. NOT started as a task; the prototype itself is mid-build across 16 commits (d741b61 latest), fully additive so far (no file in src/engine/, api.py, or the Architecture page's documented pipeline touched). Why this exists (plan dated 2026-09-24, session_status/2026-09-25.md:123-141 — moved here because it's a load-bearing decision, not short-lived working memory): the main agent (src/engine/completion.py + the free-form tool-calling loop) was producing thin/wrong gpt-oss reports (median ~470 words, missing comparison tables, wrong-paper citations) and the diagnosed cause was the free-form agent itself (98 delegate_tasks rejections observed, retries adding no new evidence), not the search/retrieval underneath it. The prototype replaces the agent's control flow with a deterministic pipeline for the retrieval half: one schema-constrained LLM call to plan facets, code-driven search/fetch (no model tool-choice), then non-LLM extraction (BM25 -> CPU cross-encoder -> rule-based entity/jurisdiction/recency/dedup filters -> specificity rerank), with a single LLM judge call per facet at the end. Only two LLM calls per query total, versus the main agent's continuous tool-calling loop — this is the intended slow/inaccurate fix, not a rewrite of the whole agent. Original go/no-go bar (superseded 2026-09-29, see below): quote-match rate >=80% AND >=80% of facets covered by >=2 distinct sources, on 2 of 3 test queries. Actual evaluation drifted from it: heldout2 (eval/heldout2_results.md) and heldout3 (eval/heldout3_results.md) instead graded per-query as OK+/OK/OK-/WEAK by reading quotes per facet. Reconciling the two exposed that the original "quote-match rate" (kept/emitted) had silently changed meaning when the extractor moved from llm (emitted = LLM-proposed spans, a small number) to the shipped ce default (emitted = every raw BM25-reranked candidate sentence pre-cross-encoder, thousands) — the field name survived the architecture pivot, the measurement didn't. Recomputed on heldout3 (2026-09-29): it reads 1.0-1.7% across all 9 queries, structurally incapable of ever approaching 80% because the denominator is now the wrong pipeline stage.

  Evaluation bar, corrected via Es et al. 2024 ("Ragas: Automated Evaluation of Retrieval Augmented Generation," EACL 2024, arXiv:2309.15217 — read in full, 8 pages, pdfinfo-confirmed, papers/ragas_2309.15217.pdf), replacing the drifted metric above rather than picking a threshold arbitrarily:

  • Coverage (facets with >=2 distinct sources) — unaffected by the drift, directly comparable to the original bar, already computed in every run's metrics.json. Heldout3: 8/9 queries at 100%, 1 at 80% (h14), average 97.8% — clears the original 80%-on-2/3-queries bar.
  • Context relevance (Ragas §3: "number of extracted [i.e. kept] sentences / total sentences in the context handed to generation") is what "quote-match rate" was trying to measure, but Ragas scopes the denominator to the already-retrieved context pool, not the whole corpus's raw candidate sentences. Rescoped to the equivalent stage in our funnel (kept / candidates actually shown to the judge, capped at JUDGE_N=20): heldout3 gives 50.7% overall (39-65% per query) — a real, comparable number, whereas the old field's 1-1.7% never was. No fixed pass threshold exists yet (Ragas itself proposes none; it validates correlation with human judgment instead — see next point), so this is tracked, not gated, until a threshold is chosen against real data.
  • Faithfulness (Ragas §3: fraction of the answer's claims inferable from context) is structurally satisfied by construction here, not something to keep measuring: every kept quote is verified as a literal substring of its source (quote_in_source) before it's ever counted. Ragas built this metric to catch a free-generation LLM inventing claims; this pipeline never generates prose in the extraction step, so the failure mode it targets can't occur here. Worth stating as an architectural property in the Architecture page if/when this pipeline gets a real architecture section, not as an ongoing eval line item.
  • Precision/recall against the hand-labeled Spanish relevance sheet (already used for extractor tuning, P0.67/R0.68 — CLAUDE.md) is the metric that's actually methodologically sound here, being validated against real human labels the same way Ragas validates its own metrics against the WikiEval human-judgment dataset (0.70-0.95 agreement in their Table 1). This should be the primary "is extraction actually working" gate going forward, not a self-referential kept/emitted ratio.
  • Answer relevance (Ragas §3: embedding similarity between the question and LLM-generated reverse-questions from the answer) has no current equivalent in this pipeline and isn't being added now — noted as a real option if facet-level "does this quote actually answer the question" checks are ever found lacking, not because a gap was found.

  Before any merge decision: (1) pick or validate an actual pass threshold for context relevance against a fresh held-out set, since heldout3 only gives a descriptive number today; (2) name the actual integration shape (replace the agent's retrieval step, become a fast-path alternative surface reachable from CLI/TUI/API, or feed its evidence table into the existing agent as a tool) — currently undecided, not even informally; (3) run the standing TUI/CLI/API parity check and an Ollama-independent path check (project rule) once an integration shape exists, since python -m pipeline.run is the only entry point today.

  Two known-open bugs from live held-out runs have been fixed and live-validated (2026-09-29, real data, not just unit tests — see session_status/CURRENT.md for the replay methodology): jurisdiction filter missing a ccTLD signal (h08 — confirmed live against the real adres.gov.co page: False before the fix, True after) and "Paxos" matching "Paxos Commit" (h12 — replayed h12's saved sources through the fixed judge prompt with the real deepdelve-gpt-oss model: all 8 explicit "Paxos Commit"-named quotes that were previously kept are now excluded from f2/f4; one residual quote remains that discusses Paxos Commit's mechanism without naming it in the sentence itself — an inherent limit of sentence-level judging, not something the prompt fix could reach).

  Held-out set 4 (h19-h27, eval/heldout4_results.md, 2026-09-30, fresh data): 0 OK+ / 3 OK / 5 OK- / 1 WEAK vs heldout3's 2/1/4/2, so the post-freeze fixes did not clearly generalize (grades provisional: 3 quotes/facet read, single reader). Root causes found by replaying saved sources (details session_status/CURRENT.md): named entity dropped because query_entities fused "Ghana's GDP" and sentence-initial "Compare Finland's" into discarded title runs (so the re-plan guard never fired; the planner itself returns the third entity when hinted); _COUNTRIES["eu"] lacked bare "EU" (a facet's 60 CE-positive candidates all died at the entity filter); region entities ("Asia") matched literally (UN M49 membership now used for the 42 known countries). Fixed and measured on 24 hand-labeled multi-entity queries: entity recall 0.79 -> 0.86. Still open: judge accepts any numeric fact about the topic (Criteria-Based LLM Relevance Judgments, arXiv:2507.09488, read in full: per-passage kappa 0.3, 92.2% of disagreements too lenient; a prompt change must be validated on the hand-labeled relevance sheets, not the facet it was tuned on), retrieval scarcity for some facets (verified for Finland adoption), and lowercase concept entities. A previously claimed judge non-determinism was NOT reproduced (8 repeats + 2 processes identical) and is withdrawn. Live rerun of h23-h25 at fad56df (no popups, virtual display): h25 now has a Ghana facet, h24's first facet went from 0 to 8 sources. Further root cause, verified by counting: select_ce cut the CE top-40 BEFORE the entity filter, so entity facets spent their pool on higher-CE wrong-entity sentences (Asia: 6 survivors vs 25 entity-first); fixed (filter, then cut) and A/B'd on saved h23 sources with the judge: Asia 1 -> 6 kept quotes, Africa 1 -> 2, Europe 3 -> 9. Stemming (Porter/Snowball) was tested and does NOT help (Asia 53 -> 53, Africa 13 -> 15 of 25 targets reach the BM25 pool; arXiv:2402.11757 read in full, English doc-retrieval evidence only) and neither does raising the per-source BM25 cap. Facet-completeness verification is supported in principle by arXiv:2609.35774 (read in full; different stage, closed corpus) but not built. Evidence-path smoke test (CLI, h24 evidence, 2026-09-30): loaded 47 findings and wrote a draft with real page titles for all 27 sources, but never converged: find_cross_source_contradictions (src/utils/grounding.py) returned 13 hits on the run's own workspace, 12 of them years (e.g. "EU AI Act" 2024 in force vs 2021 proposed / 2023 from ISO/IEC 42001:2023 text near the name) and one 7\u202f% vs 7%, and its directive forces the Builder to state the other figure. Fixed: year-kind figures are never compared, and figures are compared with whitespace removed; the designed percentage case still fires (verdict-matrix row and a new scenario in test_grounding_checks_core.py). FEVER (Thorne et al., arXiv:1803.05355, read in full, 20 pp) is a claim-verification dataset, NOT a cross-source-disagreement method, so the "FEVER-style" label on this check is loose; the paper supports only the principle used here: refutation requires a single-valued attribute (guidelines A.7.1/A.8: "place of birth can only be one place", "Multiple citizenships can exist"), and its own contradictions (0.52%, n=5) are attributed to entity-resolution errors or inconsistent pages. Not addressed: nearest-figure pairing still has no ownership limit for non-year figures, no corroboration exemption, and the verdict surfaces only hits[0]. Why that smoke run took 51+ min (Ollama journal, 57 requests): 48.2 min generating vs 1.6 min prompt eval; 5 requests hit the 12000-token max_generation_tokens cap (3.5-4 min each, 18.4 min) and 13 more made 3k-12k tokens for ~1.3k-token documents, i.e. mostly hidden reasoning. Raw /api/chat probe on deepdelve-gpt-oss (n=1 per setting): think:false (what orchestrator.py:766 sends for enable_thinking: false) did NOT suppress reasoning (4481 chars, 1536 tokens for a 321-word answer); only think:"low" did (15 chars, 612 tokens, 2.7x faster). The one Builder unexpected end of JSON input failure ended at 19:14:37 with a 12000-token generation, 2 s before attempt 3 was recorded: consistent with the cap cutting a long reasoning+tool-call generation mid-JSON (correlation, raw not captured). Deadline: run_completion_check deliberately extends budget_deadline by max_run_minutes/max_attempts (~4.5 min) per NEW problem type, up to 4x, and checks it only between dispatches, so a 45-min budget legitimately ran ~54+ min. Open decision (not changed): send think:"low" for gpt-oss. Primary-source research on gpt-oss thinking control (2026-09-30, all read in full): OpenAI gpt-oss model card (arXiv:2508.10925, 35 pp; papers/): the model is trained for exactly three effort levels set by a "Reasoning: low|medium|high" system line (§2.5.2), there is NO off mode, and accuracy scales log-linearly with CoT length (§2.6.1, Fig 3). Table 3 for gpt-oss-20b (our model) low/medium/high: Tau-Bench Retail (function calling) 35.0/47.3/54.8, Tau-Bench Airline 32.0/42.6/38.0, SWE-bench Verified 37.4/53.2/60.7, GPQA 56.8/66.0/71.5, AIME25+tools 57.5/90.4/98.7 => low is NOT free for tool use (about -12 pts vs medium on Tau-Retail); benchmark numbers, not this project's task. Ollama thinking docs (docs.ollama.com/capabilities/thinking, whole page): /api/show lists a model's valid values (gpt-oss: ['low','medium','high'], default 'medium'; confirmed on deepdelve-gpt-oss, Ollama 0.34.4); false = "request no thinking output, if the model permits it". ollama/ollama#12589 (whole issue): the gpt-oss template emits Reasoning: <level> only when think is a level or unset; with think=false NEITHER branch fires, so no Reasoning line is sent at all (identical in our model's template). So false is an untrained configuration, not "off". agent_framework_ollama forwards a string level unchanged (verified via _prepare_options; its think: bool annotation is typing only). Why the main path sends the boolean (traced, not guessed): bool written 2026-07-28 (5cbbb04) for the Ornith bake-off from the framework's annotation; the 2026-07-21 note judged gpt-oss's un-disableable reasoning harmless because it lands in a separate field (a content-pollution criterion, token cost not considered); the 2026-07-26 vLLM smoke test only showed enable_thinking is "harmlessly ignored"; on 2026-09-25 the stage-graph pipeline found "think=False does NOT help on gpt-oss" and moved to "low" at ONE call site (src/pipeline/run.py), recorded only in gitignored session notes, never propagated to src/engine/orchestrator.py (the sibling-surface failure this project's CLAUDE.md warns about); all 8 pages of the wiki literature review (RESEARCH.md's target) contain thinking-control research only for Qwen3 suppression and Ornith, none for gpt-oss effort levels. The baseline model was never put through the Model Evaluation Standard's point 1 (raw API test that the operating mode reaches the model), since that point is scoped to candidates. The orchestrator's 12000-token cap comment attributes runaway generations to quantization EOS degradation; effort level is not mentioned. Untested lead: Harmony guidance (§2.5.1) says earlier turns' reasoning traces should be dropped from multi-turn history, and agent_framework_ollama appears to re-send thinking on assistant messages (_chat_client.py:493); whether that costs context/time here has not been measured. Re-test after the Ollama upgrade 0.34.4 -> 0.40.9 (2026-10-23, raw API, served model deepdelve-gpt-oss, runner llamacpp, 100% GPU, num_ctx 16384): /api/show still lists think values low/medium/high (default medium) and the template is unchanged (Reasoning: line only when think is a level or unset; think=false emits none). One fixed question (answer 79, 3 repeats native, 2 compat), median generated tokens / thinking chars: unset 462/1353, false 632/2017, low 421/583, medium 741/1432, high 2088/5611; compat reasoning_effort low 533/692, high 2218/6285. All native runs correct (3/3 each); compat low 1/2 (the 138-token run was wrong). So false is still NOT off (632 tokens, thinking present), levels order low < medium < high, and compat reasoning_effort maps to the same levels. Limits: n=3 cannot separate unset/false/medium (unset and medium send the identical prompt yet differ 462 vs 741, which is the noise floor); only low vs high is resolved; no like-for-like 0.34.4 numbers exist on this prompt, so "unchanged" rests on the template and qualitative behaviour, not a paired comparison. Not re-tested: tool-call serialisation bugs, repeated-token abort, other models. Recommended next step (not done, needs a decision): A/B think:"low" vs the default per role on the project's own eval, not a global flip. Other-model re-test on Ollama 0.40.9 (2026-10-23, native /api/chat, one model loaded at a time, nested-array delegate_tasks-style tool, 4 calls each at temperature 0): mistral-nemo:12b, llama3-groq-tool-use:8b, llama3.2:3b all returned a proper list of dicts 4/4, so the nested-array stringification found earlier did NOT reproduce here (caveat: one simple prompt, temperature 0 makes the 4 calls near-identical, so n is effectively 1 per model; a fix upstream is plausible, not proven). ministral-3:8b: same tool 4/4 OK, and a multi-line write_workspace_file content argument (the raw-newline shape behind the open JSON bug) 6/6 OK at temperature 0.08 with a short context; the bug previously fired in 3 of 5 FULL runs, so this clears a probe, not the model (a full run under the Model Evaluation Standard is still needed before the BLOCKED status can change). qwen3:4b: think=false still generates the SAME number of tokens as think=true (1592 vs 1592 on a trivial multiplication) and the reasoning appears in the visible content instead of the thinking field, so the Qwen3 think passthrough bug is STILL PRESENT on 0.40.9. Repeated-token abort guard: not testable without a reproducible degenerate loop, no regression seen in the two 0.40.9 agent runs. Probe: scratchpad toolprobe.py. A/B on the same fixed evidence (h24, eval/thinking_effort/run_ab.sh) found two harness defects before any verdict: baseline_1 aborted at ~7 s (Ollama repeat guard), the writer dispatch exception fell through to the Planner, which then did free research (5 delegate_tasks, 11 searches, no report after 31 min, stopped by hand); low_1 finished in 7.4 min and passed the project's own checks (n=1 each, NOT a verdict). Both fixed 2026-09-30 (see the Architecture page section 2, "Evidence-only runs and transient Ollama aborts"): transient-abort retry classification, and a structural evidence_only lock in check_quota. The A/B is being rerun on the fixed code. Third defect found in the same A/B (fixed 2026-09-30, utils.grounding.snap_urls_to_fetched, applied in write/edit_workspace_file): every run's first findings.md draft was rejected (findings_ungrounded, a wasted writer+reviewer cycle) because gpt-oss damages URLs while WRITING: swapped non-breaking hyphens (U+2011), spliced hyphens plus zero-width spaces (e-u-ai-act), ellipsis truncation. Measured: the evidence handed to the writer has none of these characters, accepted artifacts have none inside URLs, the writer's own prose has hundreds (U+2011 258, U+202F 246 across 4 findings.md); raw probe n=5 per setting: copy-only tasks are exact (0 corruption), findings-writing produced U+2011 in 14 of 15 runs at think false/low/medium alike, so the effort level is NOT the lever. Repair is evidence-based (snap to a fetched URL only on a unique letters-and-digits skeleton match, or a unique prefix for an ellipsis); real en-dash URLs, ambiguous and hallucinated URLs are untouched. Offline replay of the 3 real rejected drafts: 2 now pass; the third has a looped garbage URL (e-e-e-e...) and stays rejected. Lead, NOT established: that looped-hyphen degeneration is the same family as the Ollama repeat-guard abort. The writer's prose typography (U+2011/U+202F) is deliberately not rewritten. Lead tested (2026-09-30; scripts eval/thinking_effort/{guard_mechanism_probe,logging_proxy,replay_writer}.py, results replay_writer.jsonl). (1) On our server (Ollama 0.34.4) the repeat guard aborts at exactly 101 identical consecutive trimmed stream events (repeated '-', 'a', blank lines; asking for 500 U+2011 made the model drift into soft-hyphen/U+2051 runs and abort 2 of 3); an ALTERNATING loop ("e-e-e-", the garbled-URL shape) never trips it and runs to the token cap. So the lead as first stated is refuted for the garbled URL. (2) Replay of the REAL captured FindingsWriter request (proxy capture, 9.4k-token prompt, temperature 0.2, num_ctx 65536), n=10 per setting: think=false: 0 guard aborts, 5/10 hit the 12000-token cap and never wrote the file (median 11.6k tokens, 213 s; longest visible identical run 43, an ellipsis token); think=low: 1 guard abort (rep 4, after 44 s and only 18 client-visible events), 0 cap hits, wrote the file 9/10 (median 3.4k tokens, 59 s). The abort is real, reproducible from the real request, rare, and NOT specific to false; it fired while the tool-call arguments (the file content) were being generated, which Ollama does not stream, so the repeated token is NOT identified (needs the raw llama-server stream). (3) 9/9 low files contained U+2011 and 20 URLs had odd characters across them, so the URL repair is needed with low. Raw sweep (2026-09-30, deepdelve-gpt-oss / Ollama 0.34.4, eval/thinking_effort/raw_sweep.py, n=3 per cell, 30 calls, 0 errors), tokens median (min-max): 250-word prose: false 3915 (2545-12000, 1 of 3 hit the 12000 cap), omitted 4695 (1279-5181), low 696 (688-886), medium 1530 (717-2053), high 8878 (4781-12000, 1 capped); wall median 62 / 75 / 11 / 24 / 144 s. Tool call (write_workspace_file, must name notes.md with >=40 words): false 224, omitted 1167, low 169, medium 1207, high 2514 tokens, 3/3 correct at EVERY setting. So false is not off and not stable on long-form turns (2.5k to 12k tokens, one cap hit), low is ~5x cheaper and steady there, and short tool-call turns are cheap and correct under both false and low. This is a micro-benchmark, not a quality result; the system-level A/B (same evidence through writer/builder/reviewer, settings.thinking_effort: low vs baseline) is the test that decides. New lever settings.thinking_effort (orchestrator.py, native Ollama only, unset = unchanged, typo raises; test in test_grounding_gates_and_quotas.py). Separately, the live config's enable_thinking kept flipping to true because test_routing_cache_and_misc.py's palette scenario ran the real /toggle_thinking -> config.save_config(); fixed by stubbing save_config in that scenario (suite now leaves the file byte-identical).

• Completion-check escalation ladder is model-capability-agnostic, raised wiki page Research-Small-Model-Reliability Finding B (2026-08-27) — CLOSED 2026-09-11. CONSECUTIVE_SAME_PROBLEM_ESCALATION_THRESHOLD (engine/completion_starvation.py, default 3) is the one shared number _capped (every non-self-resolving completion check) and _compute_force_whole_rebuild's escalation both use to decide when a run has re-litigated the same problem enough times to bail into salvage — previously a hardcoded module constant, so every model got the identical 3-strike budget regardless of whether its own run history (completion_check_attempts) already showed it wasn't converging. Added get_escalation_threshold(), which reads settings.completion_check_escalation_threshold (default unchanged at 3) — both call sites now go through it instead of the bare constant. A small-model eval config can set this lower (e.g. 1) so a known-weaker candidate reaches an honest acknowledged-gap/salvage report sooner instead of burning its whole max_run_minutes budget mid-correction, which live _run_state.json data from the qwen3.5-9b/hermes4-14b-fix/ granite4.1-8b bake-off (2026-08-28/29) confirmed is exactly how those runs were failing: time_taken clustered at the configured max_run_minutes (1080s) with completion_check_ attempts still in progress, not a converged or even a fully-exhausted retry budget. New test: test_completion_starvation.py's _escalation_threshold_override_scenario. This does not by itself fix small-model reliability — it only lets a config trade completeness for survivability; RESEARCH's other findings (A, already fixed as a side effect of the run-lifecycle unification below; C, D) remain informational, not actioned.

• TUI's client-side tool execution never awaits an async tool, found live 2026-09-11 — CLOSED 2026-09-11. In engine/tui.py's _resolve_tui_approval_requests (the interactive-approval tool-execution path, reached only when settings.permissions marks a tool "require_approval" — off by default, which is presumably why this went unnoticed): result_str = str(tool_func.func(**args_dict)) called the tool's underlying function directly with no await. Harmless for a sync tool, but web_search (tools/web.py) is async def — confirmed live via a real Pilot-driven TUI run with settings.permissions = {"web_search": "require_approval"}: clicking Approve produced RuntimeWarning: coroutine 'web_search' was never awaited and a stringified coroutine object ("<coroutine object web_search at 0x...>") instead of real search results, fed straight back to the model as a tool result. Confirmed pre-existing (not introduced by the 2026-09-11 run-lifecycle unification) via git show HEAD~2:src/engine/tui.py — both of run_agent's former near-duplicate approval blocks (now merged into this one helper) had the identical unawaited call. Fixed by branching on asyncio.iscoroutinefunction(tool_func.func) (tui.py:731-735), the same pattern already used by tools/core.py's with_quota decorator — await the async case, call plainly otherwise. Regression test added: test_resume_and_tui_qoe.py's _async_tool_approval_scenario, which registers a fake async tool in WORKSPACE_TOOLS, drives _resolve_tui_approval_requests directly with config.AUTO_APPROVE = True, and asserts the returned tool-result Content holds the real string, not a stringified coroutine — confirmed to fail with the pre-fix code (via git stash) and pass with the fix.

• create_local_agent's nested-closure god-function — CLOSED 2026-08-24. _run_single_task and delegate_tasks were deeply nested closures inside create_local_agent (1098 lines), capturing dozens of enclosing locals by reference, with zero direct test coverage. Resolved 2026-08-23/24: four characterization-test layers landed first (pre-dispatch validation, the streaming/retry loop, post-loop grounding/finding-storage, and the scope-relevance check — see test_structural_checks.py's _create_local_agent_characterization_scenario, _run_single_task_streaming_characterization_scenario, and _run_single_task_post_loop_characterization_scenario), catching one real previously-unknown UnboundLocalError on Message along the way (root cause: three sibling branches each redundantly re-imported Message locally, which shadows the module-level import for the whole function — fixed by deleting the redundant imports, not adding a fourth). Then the actual decomposition: both closures' bodies moved to module-level _dispatch_single_task/_dispatch_tasks_batch, every implicit closure capture turned into an explicit parameter; create_local_agent now just builds thin same-named forwarding closures (preserving mutual late-binding and every external call site unchanged). create_local_agent: 1098 → 184 lines. All four characterization layers pass unchanged against the decomposed code, plus test_tools.py and ruff check .. Full narrative belongs in the wiki's Completed page on the next migration pass, not repeated here.

• completion.py's mixed responsibilities, scoped 2026-07-29 — five slices extracted (2026-08-24), only run_completion_check and the routing tuples deliberately still untouched. The file's own header describes it as a clean list of pure Ctx -> Optional[Verdict] check functions, but it also contains findings- authoring/evidence-assembly logic, disk-touching quarantine/restore/salvage helpers, async sub-agent dispatch orchestration, the task-verification ledger mutator, and the completion-check state machine tying all of it together — none individually bug-prone, but "add a new completion check" requires understanding all of the above living in one namespace, and the Architecture page §1 documents real, delicate cross-cutting invariants here (four routing tuples that must all agree, the starvation/capping machinery) that a 2026-07-24 session hit five real bugs in, in one sitting. Slice 1 (quarantine/restore/salvage): _quarantine_artifact, _content_unchanged_since_last_quarantine, _restore_quarantined_draft, _salvage_narrated_report

  • its two banner constants, _ensure_writer_quota_headroom, _ensure_reader_quota_headroom, _is_transient_ollama_json_error, _dispatch_task_retrying_transient_json_error moved to src/engine/artifact_salvage.py — zero dependency on Ctx/Verdict or the check-list/starvation machinery, pure disk I/O + quota-dict arithmetic + one retry wrapper. Slice 2 (group A — the 27 check_* functions + Ctx/Verdict): moved to a new module, src/engine/completion_checks.py, per the planning pass's recommended sequencing (least interconnected, best first candidate). Ctx/Verdict moved alongside the checks (every check takes/returns one) so completion.py imports them FROM completion_checks.py, never the reverse — avoids a circular import, since completion.py still owns COMPLETION_CHECKS/GROUNDING_CHECKS (left in place per the plan) and references every check by name. 4 of the 30 helpers moved too (_findings_facet_coverage, _facet_coverage, find_duplicate_report_sections, _redelegate_directive); _ablation_disabled stayed (used outside the check functions too). One real circularity found doing this: _capped/_consecutive_occurrences/_dedupe_findings/ _is_citable_finding are called by both group-A checks AND the findings-evidence-assembly/ starvation machinery that stayed in completion.py — a module-level import back from completion_checks.py would fail (those names don't exist yet when completion.py's own top-of-file import of completion_checks runs). Resolved with a local (function-body) from engine.completion import ... inside the 8 specific check functions/helpers that need them (check_thin_coverage, check_task_verification_flagged, check_findings_underuses_ evidence, check_report_underuses_evidence, check_propagated_ungrounded_content, check_stale_findings, _findings_facet_coverage, _facet_coverage) — deferred to call time, after both modules are fully loaded, so it's safe; every other check function needed no changes. completion.py re-exports Ctx/Verdict/all 27 check functions/find_duplicate_report_sections/ _findings_facet_coverage/_facet_coverage at module level (the same re-export pattern as slice 1's _restore_quarantined_draft via engine.tui), so every existing external import path (test_structural_checks.py's direct from engine.completion import check_thin_coverage, Ctx etc., finetune/*.py's similar imports) kept working with ZERO test/finetune-script edits needed. Slice 3 (group B — findings evidence-assembly): _is_citable_finding, _verification_warning_targets_url, _update_task_verification, _dedupe_findings, _collapse_multi_url_task_findings, _uncited_task_names, _reorder_findings_for_position_bias, _find_propagated_bad_content, _build_findings_source_material + their 5 dedicated constants (_CUTOFF_ONLY_SUMMARY_RE, _WARNING_MARKER_RE, _VERIFICATION_WARNING_BLOCK_RE, _VERIFICATION_FLAGGED_URLS_RE, _NEAR_DUP_FINDING_OVERLAP_THRESHOLD) moved to a new module, src/engine/findings_evidence.py. No circular import this time — none of these 9 functions take Ctx/return Verdict or call _capped/_consecutive_occurrences, so the new module imports _is_null_finding_summary (utils.grounding) and _looks_like_renamed_task/ _content_word_overlap/get_context_budget (engine.orchestrator) directly at module level, no local-import trick needed. completion.py re-exports all 9 + _CUTOFF_ONLY_SUMMARY_RE — this re-export is now load-bearing for TWO consumers, not one: test_structural_checks.py's/ finetune/*.py's direct imports (as before), AND completion_checks.py's own group-A local imports (_dedupe_findings/_is_citable_finding/etc., added during slice 2) that still say from engine.completion import ... — confirmed these still resolve since completion.py re- exports every name they need. Also removed 4 now-dead imports from completion.py's own top (_is_null_finding_summary, get_context_budget, _looks_like_renamed_task, _content_word_overlap — only the moved functions used them), caught by ruff check . F401 immediately after the move. Slice 4 (group C — dispatch orchestration): _dispatch_writer_review_fix, _dispatch_per_facet_builder_fix, _dispatch_per_facet_findings_writer_fix, _select_deepening_tasks, _dispatch_deepening_round + _WRITER_EMPTY_RETRY_ATTEMPTS/ _MAX_FACET_DISPATCHES moved to a new module, src/engine/completion_dispatch.py. One real circularity: _dispatch_per_facet_builder_fix uses _BUILDER_NO_DELEGATE_CLARIFICATION, a constant defined in completion.py right alongside _BUILDER_FIXABLE_PROBLEMS (left there deliberately — it's also used twice more in run_completion_check itself, group E, so moving it would just create the reverse problem). Resolved the same way as slice 2's circularity: a local (function-body) from engine.completion import _BUILDER_NO_DELEGATE_CLARIFICATION inside just that one function. completion.py re-exports all 7 moved names — load-bearing for test_structural_checks.py's direct imports of _dispatch_writer_review_fix, _select_deepening_tasks, and _WRITER_EMPTY_RETRY_ATTEMPTS, and for run_completion_check itself (staying in completion.py) calling _dispatch_per_facet_builder_fix/ _dispatch_per_facet_findings_writer_fix/_dispatch_deepening_round/_MAX_FACET_DISPATCHES by name. math/os/asyncio became dead imports in completion.py after the move (only these 5 functions used them) — removed via ruff check . --fix. Slice 5 (group D — the starvation/capping state machine): unlike slices 1-4, this is the the Architecture page-flagged hazard itself (5 real bugs in one 2026-07-24 sitting), so per the planning pass's own recommended sequencing this got a coverage audit before any move: _capped had no direct behavior test (only a static grep enforcing that every non-self-resolving check calls it), and _apply_starvation_yield/_collect_other_active_problems/ _with_other_problems_addendum had zero direct tests at all. Added characterization tests for all three (committed separately, 520d08c, before touching any structure) confirming the exact behavior read from source: _capped's threshold/skip_problems logic, _apply_starvation_yield's stuck-then-probe-target/no-target/target-returns-same-problem/target-empty branches, and _collect_other_active_problems/_with_other_problems_addendum's exclusion-of-the-winner/cap- at-3/no-op-when-nothing-else-active behavior. Only THEN moved _consecutive_occurrences, _consecutive_tier_wins, _capped, _yield_to_starved_check, _STARVATION_YIELD_TARGETS, _apply_starvation_yield, _collect_other_active_problems, _with_other_problems_addendum, _other_grounding_problems, _with_other_grounding_addendum + their 4 dedicated constants (_STARVATION_SKIP_THRESHOLD, CONSECUTIVE_SAME_PROBLEM_ESCALATION_THRESHOLD, _OTHER_ACTIVE_PROBLEMS_CAP) to a new module, src/engine/completion_starvation.py. No new circularity beyond the established pattern: the new module imports Ctx/Verdict/ check_report_underuses_evidence from completion_checks.py (one-directional, no cycle); completion.py re-exports everything moved, load-bearing for completion_checks.py's own group-A local imports of _capped/_consecutive_occurrences and for test_structural_checks.py's direct imports. cheap_grounding_problems (from utils.grounding) stayed re-exported from completion.py too, even though nothing there calls it anymore, because test_structural_checks.py imports it from engine.completion directly — caught by running the suite, not just ruff check ., since an unused-import lint pass alone doesn't know an external file depends on the re-export. All five slices verified by running the existing suite unmodified plus a before/after sorted-line-set diff confirming no body line was dropped or duplicated (the only line-level diffs each time are import restructuring/header comments, never code content). completion.py: 3854 → 3650 (slice 1) → 2235 (slice 2) → 1618 (slice 3) → 1148 (slice 4) → 873 lines (slice 5). New completion_checks.py: 1468 lines. New findings_evidence.py: 637 lines. New completion_dispatch.py: 493 lines. New completion_starvation.py: 304 lines. test_structural_checks.py, test_tools.py, and ruff check . all pass after each slice. Group E (run_completion_check + the four routing tuples), 2026-08-24: coverage-audited first per the same discipline as group D — found run_completion_check already has 95 references in test_structural_checks.py, including dozens of direct end-to-end _asyncio.run(run_completion_check(...)) invocations across scenarios plus source-inspection pins on specific internal details, the opposite situation from create_local_agent's near-zero coverage. Given that, the four routing tuples were left in place (they have exactly one real consumer — run_completion_check itself — plus one audit test reading them via _comp.COMPLETION_CHECKS/_comp.GROUNDING_CHECKS; moving them elsewhere would be pure reshuffling with no complexity reduction, unlike groups A-D's helpers which had genuine circular-import/logical-grouping reasons to relocate). Instead extracted the 3 genuinely separable, no-early-exit/no-continue sub-phases of run_completion_check's own ~630-line loop body into module-level helpers, all still living in completion.py itself (not a new file — they're tightly coupled to this one function's loop, so a new module would just add indirection without reducing coupling): _detect_verdict (build Ctx, run the two-tier COMPLETION_CHECKS/GROUNDING_CHECKS scan with all starvation/addendum wrappers), _compute_force_whole_rebuild (the consecutive-same-problem escalation check, including its attempt = max_attempts mutation, now returned as a tuple), _notify_final_verdict (the terminal quarantine-restore/salvage/notify branch, side-effect-only). Deliberately did NOT touch the dispatch branches (Builder/FindingsWriter/thin_coverage/report_underuses_evidence/ findings_underuses_evidence) — each ends in a continue back to the OUTER while loop, which a nested helper cannot replicate without restructuring control flow into a return-a-sentinel shape, a real behavior-change risk for a mechanical pass. One legitimate test update needed (unlike groups A-D, which needed zero): a source-inspection test asserted 'run_state.data.get("fetched_urls")' in inspect.getsource(run_completion_check) — updated to inspect _detect_verdict's source instead, since that's where the logic legitimately now lives; the sibling _BUILDER_NO_DELEGATE_CLARIFICATION count-2 assertion needed no change since that logic stayed in run_completion_check itself. Verified via ruff check . and the full existing suite (no behavior change, only 2 lines of test updated to point at the new location). run_completion_check itself: ~633 → 432 lines. This closes out the completion.py decomposition plan. No further splitting of this file is planned — run_completion_check's remaining body (the dispatch branches + final loop control) is the genuinely irreducible core, and the routing tuples correctly live beside their only consumer.

  Follow-up, 2026-09-07 (ponytail-audit finding): completion_checks.py itself had grown back to 2098 lines (from group A's 1468, as new checks landed: check_editorializing_content, check_missing_specific_item_per_facet) and was split further along the SAME COMPLETION_CHECKS/GROUNDING_CHECKS boundary completion.py's own routing tuples already draw. Three files now: completion_checks.py (shrunk to the shared core — Ctx/Verdict/ _citation_format_reminder — 76 lines), completion_checks_structural.py (the 13 COMPLETION_CHECKS members + their private helpers, 1038 lines), completion_checks_grounding.py (the 19 GROUNDING_CHECKS members + their private helpers — _facet_coverage, find_duplicate_report_sections, _redelegate_directive, etc. — 1036 lines). Pure move, verified clean along an already-contiguous physical boundary (every COMPLETION_CHECKS function was already contiguous in the file, followed immediately by every GROUNDING_CHECKS function — no interleaving to untangle).

  Deliberately did NOT touch the external import surface: test_structural_checks.py has 50+ scattered from engine.completion import ... call sites (not just one top-of-file block) and 5 finetune/generate_synthetic_*_prompts.py scripts import the same way — engine.completion is a genuine, actively-used facade, not dead re-export bloat (an earlier same-session ponytail pass flagged it as a "delete the shim" candidate before this full blast-radius count was traced — corrected before acting on it). completion.py's own import block was restructured into three clearer per-module imports but still re-exports every name under engine.completion unchanged, so all 50+ call sites needed zero edits. One direct (non-completion.py) consumer did need a fix: completion_starvation.py imported check_report_underuses_evidence straight from engine.completion_checks at module level — repointed to engine.completion_checks_grounding.

  Also applied, same session: a _gp(ctx, prefix) helper in completion_checks_grounding.py replacing 10 near-identical gp = ctx.grounding_problem; if not (gp and gp.startswith("...")): return None gates that had been hand-repeated across check_claim_unsupported/check_regulation_unsupported/check_specific_figure_unsupported/ check_quote_paraphrased/check_non_url_citation/check_stub_source/check_nli_unsupported/ check_topical_mismatch/check_editorializing_content/check_uncited_claims.

  Verified: ruff check src/ test_structural_checks.py test_tools.py, the full test_structural_checks.py/test_tools.py suites, and a direct import smoke-test of every affected module (engine.completion/completion_checks/completion_checks_structural/ completion_checks_grounding/completion_starvation/completion_dispatch/findings_evidence/ engine.tui/api.py) plus the 6 external finetune/eval scripts that import from engine.completion — all clean, no behavior change.

  Follow-up, same day: test_structural_checks.py itself split into 19 topic files. Much harder than the source-side split above — the whole 10,701-line file was ONE main() function (814 top-level statements, 199 comment-delimited scenarios), chronologically ordered rather than topically contiguous, with several shared fixtures (_SRC/_FINDINGS_OK/_SOURCE_TEXT/ RunState/Ctx/Verdict/etc.) set up once early and reused bare thousands of lines later. A Python ast-based static checker traced every cross-section name dependency (479 total) back to its defining section BEFORE any text was moved, confirming a proposed 19-file topic grouping had ZERO cross-file violations once a shared ~40-line common header (real imports + 4 literal test constants) was included in every file. test_structural_checks.py is now a ~95-line thin orchestrator calling all 19 files' main()s in original order — same filename/invocation, so CLAUDE.md's "run before commit" rule and .github/workflows/ci.yml needed zero changes. Verified via a sorted-line-set diff (every non-blank line identical, same order, across the original vs. all 19 extracted bodies), the full suite via python test_structural_checks.py (identical output, including the pass@k/pass^k summary), each of the 19 files standalone, and ruff check . (whole repo) — all clean. See session_status/CURRENT.md's 2026-09-07 entry for the full file-to-topic map and methodology detail.

• run_cli/BasicTuiAgent/api.py full run-lifecycle unification, re-scoped 2026-07-29, widened 2026-08-24, CLOSED 2026-09-11. New src/engine/run_loop.py (RunLoopSurface + run_agent_loop) is now the single shared implementation of the while has_requests: loop, the malformed-tool-call/QuotaAbortException dispatch, the budget nudge-then-cutoff mechanics, and the completion-check invocation — all three surfaces call it instead of hand-rolling their own copy. Migrated one surface per phase, each with its own tests and a live smoke test against a real model before moving to the next: run_cli (Phase 1), _run_research/api.py (Phase 2), run_agent/BasicTuiAgent (Phase 3, including deduplicating its own two near-identical approval-handling copies — the main-turn loop and its sub-agent ui_callback — into one _resolve_tui_approval_requests helper). Closes a real, previously-undocumented gap as a side effect: _run_research used to only call run_state.save() on asyncio.CancelledError, never on a generic exception, so a mid-run crash left no forensic _run_state.json; the shared loop's own except Exception wrapper now covers every surface automatically. RunLoopSurface grew a handful of optional hooks (on_stream_update, on_turn_start, on_stream_exhausted, render_retry_notice, render_quota_abort, on_malformed_give_up, get_last_assistant_text) specifically to let run_agent preserve its own deliberate, pre-existing divergences without forcing a false unification: it never re-raises an unrecognized exception (shows a generic error widget and continues with a normal, not forced, completion check instead of crashing — see on_malformed_give_up's docstring), it renders one whole stream update as a unit rather than per-Content-item (its Textual widget lifecycle spans multiple items), and it still has no wall-clock/context-char budget by design (context_budget/budget_deadline stay None, always). One live bug found and fixed mid-migration: the shared loop's exception handler needed an explicit early isinstance(e, asyncio.CancelledError): raise — without it, a surface with on_malformed_give_up set (run_agent) would have silently swallowed /stop's CancelledError into a red error widget instead of ever letting it propagate; caught by a dedicated regression test before it ever shipped. One separate, pre-existing (not introduced by this migration — confirmed via git show HEAD~2) bug found live while smoke-testing Phase 3's interactive approval path: the TUI's client-side tool execution (tool_func.func(**args_dict)) never awaits the result, so an interactively-approved tool whose .func is itself async def (confirmed live with web_search) silently returns a stringified coroutine object instead of real results — rare in practice since settings.permissions defaults to no tool requiring approval at all, but real once a tool is configured that way. Flagged, not fixed, since it predates and is out of scope for this unification; needs its own fix (await when the tool's .func is a coroutine function, call plainly otherwise). See session_status/2026-09-11.md for full phase-by-phase detail, live-test transcripts, and the exact new-hook rationale.

• RAG-augmented small model, raised 2026-07-20, SURVEYED 2026-09-11, DEFERRED pending real run data. The project's own prior "RAG failure" turned out to be a benchmark-isolation bug in a deleted exact-string-match cache, not a real RAG failure, see the wiki's Architecture Synthesis page for the full literature review. Real RAG (embeddings/chunking/vector retrieval) is architecturally different from what failed before, so the old rejection doesn't automatically block it, but any persistent cross-run cache, RAG or not, must be explicitly isolated per model during comparative benchmarking or the same contamination bug recurs regardless of the retrieval technique underneath it — the one non-negotiable constraint from this project's own history, already enforced in utils/rag_cache.py (below). 2026-09-11 scoping survey (approaches + comparable OSS projects, not yet a full-paper read — see verification note below) found the specific idea this item originally raised — chunk/embed/retrieve over a run's OWN freshly-fetched documents, mid-run, to help the small generator synthesize — is not supported by the evidence at this project's model tier, and has a documented failure mode that could make output worse, not better: Sanchit Pandey (BITS Pilani), "Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale," arXiv:2603.11513v1 [cs.CL], 12 Mar 2026 — found that for UNKNOWN questions (the model cannot answer them unaided), models ≤7B fail to extract the correct answer 85-100% of the time even under oracle retrieval (the right passage guaranteed present: 7B extracts it only 14.6% of the time, 95% CI [11.9, 17.6]; 1.5B 10.0%; 360M 0.0%) — and that for KNOWN questions (the model already answers correctly unaided), injecting retrieved context destroys 42-100% of those correct answers regardless of retrieval quality (oracle and noisy dense retrieval are statistically indistinguishable below 7B, p=0.26 at 1.5B / p=0.07 at 3B — the harm comes from the presence of context, not its accuracy). Net effect of adding retrieval is negative at every model size tested, including 7B (-3.0pp with noisy retrieval, +4.2pp only under perfect oracle retrieval that real systems can't guarantee). Error analysis of 2,588 oracle failures: dominant failure mode is "irrelevant generation" (61-100% of failures depending on scale) — the model ignores the retrieved passage entirely, not that it extracts the wrong span from it. Cross-validated on a second architecture family (Llama-3.1-8B via API, FP16) with the same qualitative pattern, so this isn't a quantization artifact of the four local 4-bit models. One n-caveat the paper itself flags and that carries into ours: the 360M model's KNOWN split is n=11 only, so its 100% figures at that end of the 42-100% range should be read with caution — the 1.5B/3B/7B range is 42-64pp and is the load-bearing part of this claim for our (larger) target-model tier. Corroborated by two comparable real projects: langchain-ai/local-deep-researcher (the OSS project most architecturally similar to DeepDelve — local Ollama models, iterative research loop) does no chunking/embedding/retrieval at all; PaperQA2, the most RAG-native project surveyed, states in its own README that sub-7B models "won't get good performance" with its RAG-reasoning step. GraphRAG/RAPTOR-style advanced patterns were excluded from consideration entirely — their validated numbers are GPT-4/8B+-only, below this project's own Model Evaluation Standard's fairness bar (point 6: a candidate must clear this project's actual context/model floor, not be judged on a bigger setup). Verification (2026-09-29): read in full (10 pages incl. appendices, pdfinfo-confirmed page count, PDF kept at papers/retrieval_utilization_2603.11513.pdf), replacing the 2026-09-11 WebFetch-abstract-only skim. All figures above were re-pulled from Tables 4/5 and the error taxonomy (Figure 3) directly, not the abstract. No correction to the original headline claim was needed — it held up — but the abstract-only version omitted the UNKNOWN/KNOWN split that the finding actually depends on, the n=11 caveat on the 360M extreme of the range, and the cross-architecture replication that rules out a quantization-specific artifact. Decision (2026-09-11, user call): do not scope an in-run RAG build now. Defer until this project has banked several more successful full runs — at that point, re-scope RAG as an OFFLINE, post-hoc process over the REAL verified/cited references those successful runs already produced (download + process into a retrieval store after the fact), not live mid-run retrieval injected into a small model's context — sidestepping the context-distraction failure mode above entirely, since nothing would be forced into the generator's live context window. This is a different shape from both the original ask and from utils/rag_cache.py's existing cross-run verified-findings cache (which already stores source URL + summary per finding, model-isolated, but is populated finding-by-finding as each run happens, not batch-built from a backlog of past successful runs) — when this is picked back up, check whether it's actually best built as an extension of rag_cache.py or a genuinely separate offline pipeline before designing either. No concrete run-count trigger was set; revisit when there's a real backlog of successful runs worth mining, not against an invented number.

• TUI QoE improvements, researched 2026-07-14 — CLOSED 2026-08-24. A framework capability survey (Textual's own source, not assumed from memory) found several likely-already-working features needing live confirmation and several unused framework capabilities not yet scoped into concrete work. Two smallest, most directly requested items (message copy button, right-click paste) shipped earlier, see Completed. Two more closed 2026-08-20: click-drag select + copy confirmed already working with zero code needed (ALLOW_SELECT is True by default on both App and every widget in this Textual version, confirmed directly via textual.app.App.ALLOW_SELECT/Static.ALLOW_SELECT, not overridden anywhere in tui.py); the command palette (ctrl+p, also on by default) got a real SlashCommandProvider wiring SLASH_COMMANDS into it, live-verified with Textual's own Pilot test harness, see Completed. Widget maximize was ALSO already confirmed working, same discipline, 2026-07-24 — test_structural_checks.py's own _widget_maximize_scenario (a real Pilot test, not a mock) pins RichLog.allow_maximize defaulting True (falls back to can_focus), Screen.action_maximize() walking up to the outermost maximizable ancestor (ToolCallWidget's Collapsible), reachable via the already-wired command palette — this entry just never got updated to say so until a 2026-08-24 doc-drift catch. The remaining three items closed 2026-08-24, each confirmed against Textual 8.2.8's actual installed source, not docs/memory:

  • Theming: App.theme/available_themes and the command palette's ThemeProvider already worked with zero code (same "already works" category as widget-maximize) — the real gap was BasicTuiAgent.CSS using literal hex values instead of theme variables, so switching themes changed nothing visually. Converted every color to Textual's theme CSS variables ($panel, $primary, $secondary, $success, $text, $text-muted, $text-success, $panel-*) — live-verified via Pilot that a real rendered background color changes when app.theme switches (deliberately did NOT assert $text itself changes — it stays similar across dark themes by design, that's correct behavior, not a gap).
  • Inline autocomplete: PromptInput now passes suggester=SuggestFromList(...) over SLASH_COMMANDS — Textual's Input has native suggester support built in, confirmed via source; this complements (doesn't replace) the existing filtered-OptionList popup in on_input_changed. Live-verified: typing /sto shows an inline /stop ghost completion.
  • Tree/DataTable for the run and file pickers: both were pure DISPLAY lists before this — confirmed there was never an OptionList selection handler anywhere in this file at all; selection only ever worked by typing the exact key text (a slugified run folder name, or a filename) and pressing Enter. The run picker (_show_run_picker) now uses a real DataTable with row selection (arrow keys + Enter), and the file picker (_show_file_picker) a real Tree grouping files one directory level deep (confirmed via get_workspace_files — a sources/ subdirectory is the standard shape) — both live-verified via Pilot for genuine keyboard row/leaf selection, not just that the widget renders. TabbedContent/SelectionList were explicitly assessed and skipped — no multi-pane or multi-select UI exists anywhere in this file to justify forcing them in. All four items landed with direct Pilot-test coverage in test_structural_checks.py (no prior test existed for any of them before this pass); test_tools.py and ruff check . also pass.

• Fine-tuning, deferred pending a viable base model size, see Stretch. The correctness gate that used to block resuming fine-tuning is confirmed clear as of 2026-08-19, a new round is blocked on a different, harder problem now, no viable small base model has yet passed a real live benchmark, see the Stretch section below for the full status and the standing "one combined retrain, never an isolated LoRA" methodology rule.

Superseded, kept as a pointer so this doesn't get proposed again

• The full 11-candidate local-model bake off re run via vLLM, and the tool_choice: required fix that depended on it. Both were scoped while vLLM was this project's serving backend candidate. That effort was closed 2026-07-26: vLLM plus bitsandbytes on ROCm proved too immature on this consumer GPU across 9 disqualified/discarded candidates, several with serving-layer-shaped bugs, and Ollama was restored as the permanent backend, see the wiki's Changelog for the full trace. Any future vLLM-dependent idea needs a fresh justification to reopen that decision, not a resumption of this old plan.

• Reference repo review candidates not adopted (STORM's persona-diversity facet discovery, GPT Researcher's SourceCurator and defensive multi-strategy parsing, host-driven shrinking-budget iteration). All real, all considered, all recorded with the specific reason each wasn't adopted in the wiki's Literature Review page. Kept here only as a pointer so they don't get re-proposed as fresh ideas.

Rejected

• Large/small model dispatcher: rejected 2026-07-11 — benchmark showed small models fail sub-agent reasoning (nemo 2/10); revisit only if a small model scores ≥5 on the Colombia rubric solo.
• Knowledge cache (any backend): rejected — poisoned benchmarks/grounding; deleted in commit 929b987; do not reintroduce.
• Bibliographic-API citation verification (Semantic Scholar/OpenAlex/Crossref/arXiv, from imbad0202/academic-research-skills): rejected as a bundled default for the academic output mode — a genuinely stronger check than DeepDelve's own fetch-based grounding for published academic sources, but adds an external API dependency (rate limits, another failure mode to handle) for a benefit that only applies to formal papers, not the market-research/general-web sources most DeepDelve runs actually cite. Revisit as an opt-in flag specifically for --style academic if that mode's own fetch-based grounding proves insufficient in practice. Re-researched properly 2026-08-17 (prompted by the day's own citation-misattribution incident, to check whether this rejected feature would actually have caught it) — 2 real, primary-source-read papers plus 3 GitHub reference implementations, not a re-guess from memory:
  • CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Text (Khajavi, Sadeghi, Adhikari, Tessier — Simon Fraser University, 2026, papers/citecheck_2605.27700.pdf, read in full) — a real, rigorous evaluation (982-citation physics benchmark with controlled corruptions, 88.7 macro-F1 / 88.9% accuracy, beats GPT-5.4/Claude Sonnet 4.6/Gemini 2.5 Flash zero-shot and few-shot baselines including their own web-search variants). Its own three-class taxonomy — EXACT, MINOR (a real, recognizable paper with corrupted metadata — wrong author/year/DOI/URL), MAJOR (unrelated or fully fabricated) — maps cleanly onto this project's own incident history: the 2026-07-13 GAVEL fake-DOI incident (below) is a MAJOR hallucination; today's citation- misattribution bug is closer to MINOR in spirit, but isn't actually the same failure class (see the disqualifying limitation below).
  • The paper's own explicit limitation is the deciding fact, in its own words (§7, Conclusion): "C ITE C HECK verifies citation existence and metadata fidelity, not whether the cited source supports the surrounding claim." This directly disqualifies bibliographic- API verification as a fix for today's actual incident — that bug was two REAL, EXISTING, correctly-fetched web pages with a claim attached to the wrong one, not a fabricated or metadata-corrupted paper. Bibliographic-API checking answers "does this cited THING exist and is its metadata right," never "does THIS specific source's content actually say what's claimed" — the second question is what find_unsupported_specific_figures/nli_unsupported_ problem (this project's own existing checks) already target, and they apply to ANY source type, not just DOI/arXiv-indexed academic papers. Confirms the original rejection reasoning was correct, now with primary evidence instead of just an inference.
  • CheckIfExist: Detecting Citation Hallucinations in the Era of AI-Generated Content (Abbonato, University of Turin, 2026, papers/checkifexist_2602.15871.pdf, read in full) — a lighter systems paper (open-source MIT-licensed tool, no rigorous benchmark numbers unlike CiteCheck), confirms the same cascading CrossRef→Semantic Scholar→OpenAlex architecture and that all three APIs are usable with NO API key on free tiers (800ms rate-limit interval used to stay compliant).
  • 3 GitHub implementations checked for real technical feasibility (not guessed): the author's own tool (zabbonat/References-Validation, 28 stars, TypeScript, live web app), Vikranth3140/Citation-Hallucination-Detection (a real AAAI 2026 student-abstract companion repo — its own README's DOI badge 10.1609/aaai.v40i48.42257 verified live against CrossRef's actual API during this research pass, resolves to the real paper, correct authors — a fittingly self-referential clean citation check), and PHY041/claude-skill-citation-checker (28 stars, a single ~720-line Python script, requests-only dependency, no API key, tested by its own author at 90% known-good/100% known-bad/100% chimeric-detection/0% false-negative — an informal author self-test, not a rigorous benchmark like CiteCheck's, but a real, small, low-risk reference implementation if this is ever built).
  • One genuinely new, low-risk scope this research surfaced, not previously considered: DeepDelve already auto-generates references.bib for every run (utils/run_state.py::build_bibliography, wired into both the TUI and GET /research/{id} /references.bib in the API). A standalone, OPT-IN, post-hoc script that verifies an already-completed run's references.bib against these free APIs (mirroring the claude-skill-citation-checker shape exactly) would add ZERO latency/complexity/new failure surface to the live agent pipeline — a real, meaningfully smaller-scoped variant of the originally-rejected "build it into the live grounding pipeline" idea. Still not started (no demonstrated need yet — no live DeepDelve run has ever produced a MAJOR-class fabricated academic citation), but the standing "revisit if academic-mode grounding proves insufficient" condition now has a concretely scoped, low-risk answer ready if it's ever triggered, instead of an open-ended "build bibliographic verification" task.
• SkyworkAI/DeepResearchAgent (reviewed 2026-07-12): a general self-evolution agent runtime (RSPL/SEPL protocol layers, RL-based prompt/solution optimizers, versioned tracing) with example agents for trading/ESG/mobile — not a deep-research-specialized project despite the name. Rejected: same reasoning as the existing "no DI framework, no plugin system" stance above: its tracing/versioning goal is already served by _run_state.json, and its optimizer/self-evolution loop is out of scope for a project explicitly avoiding RL infrastructure outside the "Fine-tuning" section above (itself now deferred indefinitely).
• Fabricated/misattributed sources caught during the 2026-07-13 3-model research pass — recorded so a future session doesn't re-trust them without re-checking: a "GAVEL: Evidence- Contract Debate with Mechanized Scrutiny" paper with a fake ACL-2026-Findings DOI does not exist anywhere (checked directly, zero hits). Separately, one of the three responses attached invented mechanisms to two real papers it likely never actually read: it claimed arXiv:2603.18000 (AgentFactory) describes a disk-quota/task_uuid workspace-isolation mechanism — the real paper is about reusable sub-agent code, no quota mechanism anywhere in it — and separately claimed a real TechRxiv paper (Piskala, Agent, Sub-Agent, Skill, or Tool?) describes a "Try-Catch- Critique" 1B-parameter tool-error classifier — the real paper is an orchestration-pattern taxonomy (tool-centric/hierarchical/decentralized), no such mechanism anywhere in it. That response's citations were <25% reliable on direct inspection; its other two ideas (cross-encoder reranking, Gemma 4 12B) happened to be individually sound but were not verified by that response itself — treat as unsourced until independently re-checked, which is what happened before either was added to "Pending" above.
• platoyaoxu/pdfdownload (reviewed 2026-07-14, user-supplied link, directly relevant given the same-day ScienceDirect/Cloudflare Turnstile investigation above): a personal Elsevier/ScienceDirect batch PDF downloader — DrissionPage opens each DOI in a real visible Chromium tab, a companion AutoClick.py subprocess does OS-level pyautogui screenshot/template-match clicking (real mouse input, not CDP-synthetic) against user-supplied PNGs of the Cloudflare checkbox and the download button, with a human physically present to solve anything the templates can't handle. Confirms our own finding from the same investigation: it's very plausibly beating Turnstile specifically because pyautogui drives genuinely trusted OS-level input events, not CDP's synthetic Input.dispatchMouseEvent — a more fundamental distinction than navigator.webdriver or Playwright-vs-DrissionPage as libraries. Not adopted, on the same principle already applied to ScienceDirect above: its entire purpose is defeating anti-bot protection to bulk-scrape copyrighted publisher content (the repo's own .gitignore excludes downloaded PDFs "copyrighted & large," so the author knows what this is) — that doesn't belong in DeepDelve's default fetch path even though the "real trusted input" technique is a genuinely interesting, confirmed data point. Secondary code-quality notes for the record, not actionable for us: no timeout anywhere in either the click-watch loop or the download-wait loop (a wrong screen resolution or an inaccessible paper hangs the whole batch indefinitely), and the images/ template folder it depends on isn't shipped in the repo, so it isn't runnable as-is.

Stretch

Fine-tuning is currently the only Stretch item. Full history, from the first ROCm/GRPO feasibility smoke test through the first trained round's own live disqualification, moved to the wiki:

→ Fine-tuning

Status as of 2026-08-19: the correctness gate that used to block resuming fine-tuning is confirmed clear. A new round is blocked on a different, harder problem now, no small local model has yet passed a real live benchmark to serve as a viable base to fine-tune in the first place, see the wiki's Model Bake off page for the current search.

Standing methodology rule, engraved after real cost: every new fine-tuning objective folds into ONE combined multi-objective GRPO retrain off the same raw base checkpoint, never a separate, isolated single-purpose LoRA. Separately trained adapters cannot be merged or stacked cleanly, a model can only deploy with one LoRA active at a time. Before starting any new round: extend the existing combined reward function with the new dimension, retrain from the same raw base with every objective's prompts combined, not just the new one, re-evaluate every dimension together on held out data to confirm no regression, and redeploy as a single new combined artifact replacing the prior one.

Still-open, literature-backed candidates for whenever fine-tuning resumes, not yet acted on:

• Verify the AXPO resampling-reward mechanism against finetune/reward.py's actual writer_role_response_reward implementation before adapting it. AXPO's own stated limitations confirm its trigger mechanism assumes a binary verifiable-outcome reward, which this project's structural "did it happen" reward doesn't have; the underlying "concentrate exploration at the sparse action boundary" insight likely still transfers, the specific mechanics don't as-is.
• Apply "Iterative Reward Calibration"'s own finding, always measure a new reward term's discriminative power before deploying it, to any future expansion of writer_role_response_reward beyond its current single structural signal. That paper's own live result was a naively dense reward costing up to 14 points versus a sparse one, on the same Qwen3 family this project tests.
• DeepResearch-Slice (arXiv:2601.03261) proposes a small, separately trained boundary-prediction head that slices retrieved text down to the relevant span before the reasoning model sees it, a real, measured fix for exactly the "distracted by spurious passages" mechanism behind this project's own recurring content-vanishes-during-synthesis pattern. Not pursued, needs training a real model, same gate as everything else here.