References

References

This is the full bibliography behind DeepDelve's architecture decisions. It used to be README.md's "References" section.

Why local models are hard for this task, briefly. DeepDelve's own model bake off (10 candidates, 9 disqualified, gpt-oss:20b the only full pass) isn't an idiosyncratic gap. A published capacity floor study found qwen2.5:14b to be the "minimum viable production" threshold for reliable tool invocation, with sub 8B models failing at 40 to 85 percent plus rates on a task that's narrower and more controlled than DeepDelve's own open ended research (Huang, Malwe, Wang, arXiv:2601.16280). Separately, two studies found that small and mid sized open weight models fail at structured tool call serialization (schema valid output, wrong content) in a way a 6,000 sample fine tuning run couldn't fix, because the failure happens downstream of anything fine tuning touches (Li, Zhang, Lv, arXiv:2606.25605; Ray, arXiv:2605.26128). MAST's 14 mode failure taxonomy (Cemri et al., arXiv:2503.13657, NeurIPS 2025) maps closely onto DeepDelve's own documented bug catalog (see the "narrate instead of write," over research, and exclusion enforcement entries in the Completed list). That's evidence these are known, published agent failure patterns, not quirks specific to DeepDelve. The full literature review, its corrections, and its still open leads live on the Literature Review page.

• Huang, Malwe, Wang. When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi Agent LLM Systems. arXiv:2601.16280. This is the capacity floor evidence for the local model selection constraint above (14B as the tested "minimum viable production" threshold for tool invocation), and it's part of why we chose gpt-oss:20b over a smaller default.
• Li, Zhang, Lv. Constraint Tax in Open Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints. arXiv:2606.25605. This confirms that DeepDelve's own code never combines a JSON schema response_format with tool availability in the same call, which is exactly the condition that causes a 0 percent tool invocation rate on every open weight model tested, including GPT-OSS-20B. Its proven Two Pass Execution mitigation independently matches DeepDelve's own Planner to Builder/FindingsWriter split.
• Ray. The Constraint Tax: Measuring Validity Correctness Tradeoffs in Structured Outputs for Small Language Models. arXiv:2605.26128. This is the originating paper for the constraint tax concept above. Its "reason free, constrain late" finding independently confirms the same design choice as the Two Pass Execution paper.
• Cemri, Pan, Yang, et al. Why Do Multi Agent LLM Systems Fail? (MAST). arXiv:2503.13657, NeurIPS 2025 Track on Datasets and Benchmarks. A 14 mode failure taxonomy that maps closely onto DeepDelve's own documented bug catalog (see Completed).
• Cemri, Cojocaru, Pan, et al. Fantastic Adaptive Taxonomies and How to Use Them (ATLAS/AdaMAST). ICML 2026 Workshop on Failure Modes in Agentic AI. multi-agent-systems-failure-taxonomy/ATLAS. This induces a domain specific failure taxonomy directly from a target system's own execution traces, rather than relying on MAST's fixed vocabulary. It's a candidate idea for mining DeepDelve's own _run_state.json history.
• Liu, Lin, Hewitt, et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172, TACL 2024. A candidate mechanism, distinct from plain context size truncation, for DeepDelve's own recurring pattern where real content silently vanishes during final synthesis.
• Li, Zhang, Zhang, Chaturvedi. Coverage based Fairness in Multi document Summarization. arXiv:2412.08795, NAACL 2025. This formalizes per cluster and per document representation checks ("Equal Coverage" / "Coverage Parity") as the standard way to catch a summarizer that silently favors one source group over another. It validates the per task binary coverage design behind check_report_underuses_evidence (the Builder stage report writing check in src/engine/completion.py) as a legitimate, cheap proxy for their more expensive NLI based metric. It also closes a live confirmed gap where a report could clear a flat citation ratio threshold while still dropping an entire query facet, simply because the surviving facet happened to have more raw sources.
• Huang, Ma, Tahir, Zhang, Xiao, Xiao. Regression Accumulation in Multi Turn LLM Programming Conversations. arXiv:2607.01855. This paper finds that 55.7 percent of multi turn LLM correction regressions happen because a later fix breaks an earlier, already satisfied requirement through incompatibility, not because the model forgot. Their validated "Verification Gate" mitigation (full re verification every turn, every failing constraint made visible) directly informed the fix for a real check priority shadowing bug (utils/grounding.py::cheap_grounding_problems, src/engine/completion.py::_with_other_grounding_addendum), where real_grounding_problem's own first hit priority chain could silently shadow a second, simultaneously true grounding problem for an entire run's retry budget.
• Kamoi, Zhang, Zhang, Han, Zhang. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self Correction of LLMs. arXiv:2406.01297, TACL. Corrected 2026-08-17. This was previously miscited here as the source of the "about 64.5 percent across 14 open models" self correction blind spot figure. On a full read, that number doesn't appear anywhere in this paper. It belongs solely to Tsui's Self Correction Bench below, which this survey doesn't even cite (we checked its full reference list directly). This paper is actually a methodology critique: it finds that many published self correction studies don't clearly define their research question or design controlled experiments, and it provides a checklist (its section 8) for what a valid self correction study needs, namely a fair baseline, a genuine "own mistake" condition, and appropriate metrics. That's real prior art for treating any single self correction number cautiously, including the ones cited in this project's own README, but it isn't itself an empirical source for the blind spot figure.
• Tsui. Self Correction Bench: Uncovering and Addressing the Self Correction Blind Spot in Large Language Models. arXiv:2507.02778, COLM 2026. Read in full, not just the WebSearch summary a first pass this session mistakenly relied on. This is the original, and per the correction above the only, primary source we found for the roughly 64.5 percent figure (testing 14 open source non reasoning models). Appendix C's per model breakdown (read 2026-08-17) shows the average hides a huge spread that's relevant to this project's own model choices: Qwen3-14B, Qwen3-32B, and Qwen3-30B-A3B, tested in non thinking mode (the same regime config_template.yaml runs in), score between 0.004 and 0.108, a near total blind spot, far worse than the 64.5 percent average implies. The paper isolates the blind spot as a genuine activation failure (models correct the identical error when it's presented externally, which rules out "doesn't know better") and finds it isn't solved by closed source frontier models either (Claude 3.5 Haiku scores 52.5 percent blind spot, Claude Sonnet 4 scores 41.4 percent). That's a useful confirmation that a frontier model failing to self correct its own prior output is a known, measured phenomenon, not an anomaly. Its most actionable finding for this project: appending the single word "Wait" after a model's own erroneous output, with no fine tuning at all, reduces the blind spot by 89.3 percent and nearly matches that same model's full reasoning mode accuracy. That's a candidate addition to _dispatch_writer_review_fix's retry instructions prepend (src/engine/completion.py, commit 4dc19bc) that we haven't implemented yet.
• Song Tae-Eun. Cross Context Review: Improving LLM Output Quality by Separating Production and Review Sessions. arXiv:2603.12123. A single author, non academic preprint (Daejeon Jungang Cheonggua Co.) with no confirmed peer review venue, read in full on 2026-08-17. It proposes session separation between production and review as a mitigation for the self correction blind spot above, and it motivated routing the two checks' fix directive through edit_workspace_file (a narrowly scoped addition) explicitly, rather than an implicit full write_workspace_file regeneration from the model's own prior draft. Its own controlled ablation (4 review conditions, F1 on 150 injected errors) is a direct, positive validation of PEER_REVIEWER_INSTRUCTIONS's existing shape, not just a general motivation. The paper's best condition, Cross Context Review (artifact only, no generation prompt, F1 28.6 percent), beat its "Subagent Review" condition (fresh session but with the original generation prompt, F1 23.8 percent, p equals 0.004). In other words, knowing the original task actively hurt review quality. DeepDelve's own PeerReviewer dispatch (_dispatch_writer_review_fix, src/engine/completion.py) already sends only "Review '{artifact}' for accuracy and coherence...", no query and no production context, which matches the paper's winning artifact only condition rather than the weaker prompt aware one. We arrived at this independently, not by copying the paper.
• Xu, Zhu, Wang, Wang, Athiwaratkun, Wang, Zou, Zhang. When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework. arXiv:2506.16411, ICLR 2026. Read in full on 2026-08-17. This paper names three distinct long context failure modes: task noise (cross chunk dependence), model noise (context length confusion), and aggregator noise, where individual facts are correct but a merge or synthesis step drops whole clusters of them. The multi facet abandonment pattern we've observed is a clean match for aggregator noise specifically, which scopes which fix family (hierarchical, divide and conquer decomposition) is the natural next escalation if the edit_workspace_file directive fix above doesn't move the needle (see the Pending section of ROADMAP.md). Its formal "Regime 2: the Silo Effect" (task noise dominates; a decomposition schema loses global reasoning "regardless of model quality," Proposition 3.1/section 3.6) is a third independent paper this project has now read that converges on the same structural conclusion as the MARL paper in part 5 of the Literature Review and the Illusion of Multi Agent Advantage paper below. DeepDelve's per facet research dispatch is the well behaved case, since the task decomposes cleanly, and the consolidation and aggregation stage (FindingsWriter, Builder) is structurally the one place error concentrates, no matter which theoretical lens you use to derive it: PAC learning sample complexity, causal MAS ablation, or this paper's fidelity loss decomposition.
• Khanal, Tao, Zhou. Beyond pass@1: A Reliability Science Framework for Long Horizon LLM Agents. arXiv:2603.29231, Northern Kentucky University, 2026-04-01. This formalizes the pass@1 versus reliability gap that this project's own fixes were, until 2026-08-17, only validated against with single run anecdotes. Pass@k (succeeds at least once across k trials) and pass^k (succeeds every single time) are genuinely different questions, and τ bench's own measurement (GPT-4o: 61 percent pass@1 but only 25 percent pass@8 on retail agent tasks) shows how far apart they can be. One of the paper's three benchmark domains is literally "Agentic Web Research" (multi step search, fetch, and synthesis), which is directly on domain for us, not just analogous, and its own methodology uses k equals 3 repeats, the same floor eval/evaluate.py's compute_reliability_summary (pass@k/pass^k reporting over the existing --runs flag) is built around.
• Mustahsan, Lim, Anand, Jain, McCann. Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation. arXiv:2512.06710, AAAI 2026. Read in full including all appendices. This proposes ICC (variance decomposed into task difficulty versus agent inconsistency components) as a more rigorous reliability metric than a bare pass@k/pass^k rate. Its most load bearing finding for this project: on the benchmark task shape closest to DeepDelve's own (GAIA Level 3, "hard open ended reasoning"), even GPT-5 only reaches an ICC of 0.629 ("moderate," not "good" reliability), and GPT-4o's ICC of 0.304 means 70 percent of observed variance is trial to trial randomness, not task difficulty. As the paper puts it, "single run results are essentially unreliable." Its own convergence analysis found that stable ICC estimates need about 32 trials for this task shape (versus 8 to 16 for simpler structured tasks), and its own closest analog to a DeepDelve shaped agent (o4-mini deep research) only managed 8 trials for cost reasons, with its own Limitations section stating that isn't enough for generalizable conclusions. This directly informs eval/evaluate.py's --runs/reliability summary feature: k equals 3 is a practical floor chosen for this project's own per run cost (20 to 70 minutes on local hardware), not a claim that k equals 3 achieves statistically converged reliability for a task this hard.
• Jwalapuram, Lin, Li, Jiao, Wang, Ming, Ke, Qin, Carenini, Joty. The Illusion of Multi Agent Advantage. arXiv:2606.13003, Salesforce Research/HKUST/ UBC/NTU, 2026. Read in full. Its critique, that automated multi agent frameworks which dynamically decide their own coordination structure per query (DyLAN, ADAS, AFlow, and similar) mostly collapse into simple single agent self consistency at 10x the cost, does not indict DeepDelve's architecture, which uses a fixed, hand designed pipeline rather than per query dynamic routing. The paper's own contrast case, "Expert MAS" (deterministic, explicit role decomposition, Python orchestrated control flow, structurally the closest match to DeepDelve's own shape), is the one architecture in the paper that wins decisively (GPT-5 goes from 57.0 percent to 96.5 percent over single agent). Its real contribution for this project is the audit methodology, not a verdict against decomposition: complexity is "architectural bloat" specifically when it's added without verified causal contribution (their own causal ablations, for example reducing decision making from consensus to single agent authority raised success by 9.4 percentage points). This project's own completion check mechanisms (src/engine/completion.py) have never been held to that standard, since each was validated only by "did today's specific symptom stop recurring," never by a controlled with/without ablation.
• Su, Wu. When Do Multi Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systems. arXiv:2602.08272, University of Hong Kong, 2026-02-10. Read in full: the main body, sections 1 through 5, though the theorem proofs in the appendices weren't read line by line. This is a PAC learning theoretical analysis of multi agent RL training, empirically validated on real GSM8K data, so it's not an inference time deployment study, but its central mechanism generalizes. Decomposing into genuinely independent subtasks scales cost down, since complexity is dominated by the single hardest subtask, while dependent subtasks introduce error propagation with a quadratic worst case penalty in agent count, confirmed on GSM8K ("SARL consistently outperforms MARL due to error propagation" in the dependent subtask setting). This maps precisely onto where DeepDelve's own bugs concentrate: the Planner's per facet research dispatch is the genuinely independent case this paper's theory favors, while FindingsWriter/Builder's consolidation stage is the dependent, error propagating junction. Every bug in the 2026-08-17 seven fix arc (evidence crowding, marker leaks, the self correction loop) occurred there, not during independent per facet research.
• Jiang, Yang, Cui, et al. Deep Research in Physical Sciences: A Multi Agent Framework and Comprehensive Benchmark (DelveAgent / PhySciBench). arXiv:2606.18648. This is the primary architecture source for the Adaptive Planning Loop, Dual Granularity Memory, and Hierarchical Reflection.
• Huang, Chen, Zhang, et al. Deep Research Agents: A Systematic Examination and Roadmap. arXiv:2506.18096
• Xu, Peng. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications. arXiv:2506.12594
• Xi, Lin, Xiao, et al. A Survey of LLM based Deep Search Agents. arXiv:2508.05668
• Del Rosario, Krawiecka, Schroeder de Witt. Architecting Resilient LLM Agents: A Guide to Secure Plan then Execute Implementations. arXiv:2509.08646 (SAP, ACM, University of Oxford; corrected 2026-08-17, previously cited generically as anonymous "Plan and Execute agentic architectures survey work." On a full read, it's actually a practitioner security hardening guide, not a survey, though its plan then execute architectural description is accurate and matches what we cited it for). It describes the established pattern the Builder plus Build to Review to Fix loop maps onto: decouple planning (decompose, can use a cheaper or more strategic pass) from execution (carries out and retries mechanically), and only re plan on genuine failure rather than every step. Its own section 7.1 describes this exact re planner node pattern, fed the original objective, the original plan, and prior step outcomes, matching DeepDelve's own adaptive planning loop shape. This was directly informed by the observation that DeepDelve's Planner conversation grows unboundedly across a run with no compaction, a documented "context poisoning" risk, and that the existing delegate_tasks mechanism already gives every dispatched sub agent a genuinely fresh context. The fix is routing report writing retries through that mechanism instead of the Planner's own conversation.
• Min, Krishna, Lyu, et al. FActScore: Fine grained Atomic Evaluation of Factual Precision in Long Form Text Generation. arXiv:2305.14251, EMNLP 2023. Read in full on 2026-08-17. This is a decompose then verify pattern (break a generation into atomic checkable facts, score each independently) that the grounding layer's line scoped checks already followed in spirit. We cite it as prior art motivating the NLI entailment check (nli_unsupported_problem) below. Its own stated limitation is directly relevant to this project's check design. FActScore measures factual precision only. It "does not penalize a model that abstains from responding too frequently or generates fewer facts," an inherent precision/recall trade off the paper's own Limitations section names explicitly. DeepDelve's completion check pipeline already pairs a precision side check (nli_unsupported_problem/grounding) with separate recall side checks (check_thin_coverage, check_report_underuses_evidence) that catch a report gaming precision by omitting risky claims, closing exactly the gap FActScore's own authors flag as unaddressed by precision only metrics.
• Goswami, Kurra. HALT RAG: A Task Adaptable Framework for Hallucination Detection with Calibrated NLI Ensembles and Abstention. arXiv:2509.07475 (author names corrected 2026-08-17 from a placeholder "Anonymous" credit; no institutional affiliation is stated in the paper itself and no confirmed peer review venue exists, read in full). This is the source of the "layer NLI entailment on top of lexical/term overlap checks, don't replace them" design choice for nli_unsupported_problem. HALT RAG's own ablation (their Table 2) quantifies this directly: removing the entailment signal costs 4.5 F1 points, removing the contradiction signal costs 2.1, and removing lexical features costs 1.3, confirming NLI and lexical signals are complementary rather than redundant. That's why the entailment check only runs on claim lines that already passed the existing term overlap check, rather than gating independently. One honestly scoped limitation relevant to DeepDelve's own use: the paper was evaluated only on HaluEval's clean, relevant source documents, and its own Discussion names robustness to noisy or irrelevant retrieved context (DeepDelve's actual situation, real fetched web pages rather than a curated benchmark) as "an important and unevaluated area for future work."
• Rasheed, Banerjee, Mukherjee, Hazra. From Fluent to Verifiable: Claim Level Auditability for Deep Research Agents. arXiv:2602.13855 (title corrected 2026-08-17; previously cited under a fabricated title, "Claim Evidence Provenance in Grounded Generation," where the author and substance were right but the title wasn't, caught on a full read). A perspective or position paper with no confirmed peer review venue, it proposes the "Auditable Autonomous Research" (AAR) standard, formalizing claim nodes with typed edges to source nodes so a claim's provenance can be checked independently of its neighbors on the same line. That claim node framing, together with FActScore above, motivated decompose_claim_segments's atomic claim decomposition: splitting a line into per citation segments so each claim is checked only against its own bound citation, closing a gap where two distinct claims sharing one line could pass on a shared generic term.
• Thorne, Vlachos, Christodoulopoulos, Mittal. FEVER: a large scale dataset for Fact Extraction and VERification. NAACL 2018, fever.ai. This is the FEVER style claim and evidence framing behind find_cross_source_contradictions, which cross references a report's claims against all of its own fetched sources, not just the one cited, catching a report that silently picks one side of a real disagreement between two of its own sources without flagging the conflict.
• Santos, Peng, Macdonald, Ounis. Explicit Search Result Diversification through Sub Queries. ECIR 2010. This is the source of the greedy marginal aspect coverage reranking pattern behind tools/web.py::_diversity_rerank: after the engine's own top result, each subsequent pick is whichever remaining result adds the most new aspect terms, rather than following raw engine rank.
• Anthropic. How we built our multi agent research system. anthropic.com/engineering/multi-agent-research-system. This is independent confirmation that a multi agent research architecture (a lead agent delegating to parallel subagents with their own fresh context) beat a single agent by 90.2 percent on Anthropic's own internal research eval. It validates DeepDelve's existing Planner/Searchers/Analyzers shape, and it directly informed the decision behind the Builder plus Build to Review to Fix loop: route report writing retries through a fresh context sub agent dispatch rather than the Planner's own growing conversation.
• kyuz0/deep-research-agent: the base architecture DeepDelve was forked from.
• CYC2002tommy/Deep-Research-Agent: the source of the "full text reading is mandatory" idea and the content level claim grounding ideas.
• nashsu/llm_wiki: the source of the findings.md to final_report.md two pass pattern and the structured run state idea.
• dzhng/deep-research: the source of the schema forced FOLLOW UP DIRECTIONS idea (Searchers must return next round research leads for the Planner) and the information density rule for findings (entities, exact metrics, dates). Its structural iterative deepening loop (learnings conditioned query generation with geometric narrowing) is a ROADMAP candidate.
• Alibaba-NLP/DeepResearch (Tongyi DeepResearch): the source of the heavy search mode (test time scaling, credited in tools/web.py) and the DocumentAnalyzer verbatim evidence rule (its visit tool extractor separates verbatim evidence from summary). Its context budget endgame and the Tongyi DeepResearch 30B A3B model itself are both ROADMAP candidates.
• imbad0202/academic-research-skills: reviewed for its literature review paper structure and its Anti Leakage Protocol ("Knowledge Isolation Directive": prefer session materials over parametric memory, flag [MATERIAL GAP] instead of fabricating). Both are ROADMAP candidates for the academic output mode work. Its bibliographic API citation verification (Semantic Scholar/OpenAlex/Crossref/arXiv) was reviewed but not adopted, see the Rejected section of ROADMAP.md.
• SkyworkAI/DeepResearchAgent: reviewed for its self evolution agent runtime (RSPL/SEPL protocol layers, RL based prompt and solution optimizers, versioned tracing). Not adopted, see the Rejected section of ROADMAP.md.
• froggeric/Qwen-Fixed-Chat-Templates: a community patched Qwen 3.5/3.6 family chat template that fixes an "empty <think> block poisons the model into tool looping" defect in the stock template, plus KV cache invalidating whitespace churn across turns. Applied directly on 2026-07-28 to an Ornith-1.0-9B GGUF's tokenizer.chat_template metadata via gguf_new_metadata.py --chat-template-file (no tensor rewrite) during a live bake off, and it measurably reduced task splitting and looping regressions. See parts 4 and 5 of the Literature Review for the full before and after evidence.
• deepreinforce-ai/Ornith-1 GitHub issues #4 and #16: the model author's own issue tracker, read directly as primary corroboration that the tool looping pattern hit during this project's own Ornith-1.0-9B bake off reproduces independently across unrelated harnesses (VSCode/Continue, Claude Code). That confirms it as a real, model family wide trait rather than a DeepDelve specific bug, and it names the same root cause (froggeric's template fix, HF discussion #42 on the 35B GGUF) this project independently arrived at the same day.
• microsoft/agent-framework's own packages/devui reference app (frontend/src/components/features/workflow/execution-timeline.tsx): the source of src/static/index.html's per agent run timeline, one collapsible entry per sub agent dispatch with a running or done status icon, replacing an earlier flat interleaved log. We checked this directly against the actual framework DeepDelve is built on, rather than improvising it, after a live UX complaint that the flat log read as noise once more than one or two sub agents had run.
• assafelovic/gpt-researcher's Next.js frontend: two ideas we checked directly and adapted rather than copied verbatim. components/research/ResearchContent.tsx's disabled={loading} pattern (every input locks during a run, not just the submit button) confirmed a live UX bug in DeepDelve's own web UI. components/ResearchBlocks/elements/SourceCard.tsx is the source of the report viewer's source cards grid, with its external Google favicon service call deliberately dropped, since a per citation outbound request conflicts with this being an otherwise fully local first tool, in favor of a plain domain initial avatar.