-
Notifications
You must be signed in to change notification settings - Fork 0
Literature Review Architecture Synthesis
Part 3 of the Literature Review: the routing layer proposal, the comparative survey against other deep research agent projects, RAG reconsideration, and a novelty assessment.
Implemented and live verified 2026-07-20, full detail in Completed. The reasoning chain: the capacity floor and constraint tax literature, plus DeepDelve's own bake off, converge on small/mid models failing disproportionately at structured serialization, not semantic understanding, a failure the constraint tax paper's own SFT ablation shows fine tuning can't fix since it happens at the decoding layer. delegate_tasks decomposes into genuinely different sub problems though: semantic task decomposition needs an LLM, but routing (agent_id) is classification over 4 fixed labels, where invalid output can be made structurally impossible instead of just discouraged.
Before building, real data got checked: 1,153 real instruction/agent-id pairs extracted from 95 session logs, imbalanced but workable class counts, and 4.9% were the model inventing agent IDs that never existed, exactly the failure class a fixed-label classifier can't reproduce by construction. SetFit was considered and rejected (designed for 8-16 examples per class, DeepDelve's data is well past that regime); the choice was a frozen sentence embedding plus logistic regression, in the literature's own comparable-to-full-fine-tuning range at this data volume. Real held out results after implementation: 0.82 overall accuracy, per-class precision 0.44 to 0.89.
Prompted by a prior session where the user deliberately tested whether an unverified "most sophisticated" framing would be accepted uncritically; it was declined for lack of evidence. This is the follow up, primary sources read directly for every project already credited in References.
(arXiv:2510.24701) is a fundamentally different architecture class: a single 30.5B MoE model doing everything itself, not a multi agent system, with no published grounding layer comparable to DeepDelve's, reliability comes from RL training, not a runtime check. Not rankable against DeepDelve, different scale, different lever.
Is explicitly minimal (under 500 lines), no completion check layer, no grounding verification, no multi agent separation. Confirmed as the real architectural ancestor of DeepDelve's own iterative deepening loop (the FOLLOW-UP DIRECTIONS pattern and geometric narrowing are genuinely present in its code), but it's a comprehensibility reference implementation, not a reliability competitor.
Is a prompt driven Agent Skill, not a standalone orchestrator; its phase gating is enforced by prompt instruction, not deterministic code, and its anti hallucination mechanism is a DOI existence ping, strictly narrower than any single one of DeepDelve's grounding checks. It's ahead of DeepDelve on end to end output polish (docx/APA export, chart generation), a real, honest gap in DeepDelve's favor.
Isn't a research pipeline at all despite the name, it's a generic self-evolution agent runtime with deep research as one example app; not a fair comparison.
Is a personal knowledge base app where Deep Research is one of ~19 features, gated by a much lighter human-in-the-loop guard than DeepDelve's grounding pipeline. Confirmed as the real ancestor of DeepDelve's own findings-to-report two pass pattern. It's a far larger shipped product surface (knowledge graph, browser extension, MCP server) than DeepDelve's CLI/TUI tool.
For the narrow question of post hoc citation/claim grounding depth on a small or local model, DeepDelve is ahead of all 5 projects actually compared. Three qualifications keep this honest: different projects solve different problems (Tongyi's larger-model lever isn't available to DeepDelve's local-hardware constraint); "most sophisticated" isn't "most validated," per this project's own audit most grounding checks had never been tested against a real captured fabrication case at the time; and this is 5 projects, not an exhaustive market scan. The bounded claim: DeepDelve's verification layer is more elaborate than these 5 comparable projects for grounding depth specifically, not a claim about capability, benchmark performance, or shipped surface, where several of these projects are well ahead.
The user had a bad prior experience with RAG on an unnamed earlier project and wanted to know if DeepDelve's infrastructure would avoid the same failure. Three papers read in full before any design work.
(Garani, ACL workshop 2026, peer reviewed despite being solo authored). A structured review of 48 sources into 33 failure modes across 7 pipeline stages; 36% have no peer reviewed empirical evidence at all, and all 8 agentic orchestration failure modes are among them, meaning any RAG DeepDelve adds is "Agentic RAG" by construction and lands in the least validated part of the literature. Its core diagnostic concept, "cascade blindness" (a symptom at generation is often root caused far upstream at ingestion), is directly actionable for diagnosing the user's own prior failure. Two of its failure modes match bugs DeepDelve already found and fixed independently: recursive hallucination cascades (the narrated-but-never-written report bug class) and unbounded cost/latency spirals (the quota exhaustion loops seen in MiniCPM testing), DeepDelve's quota system is already a structural guardrail against a failure mode this paper flags as under-studied. "Position of gold bias" (strong evidence) is the same "Lost in the Middle" phenomenon cited elsewhere, independent reinforcement. Read cover to cover: its own limitations section notes single-rater grading with no inter-rater agreement, and that claims about relative frequency of failure modes rest on secondary interpretation, not direct measurement, useful as a map of the failure space, not a prevalence ranking.
(arXiv:2501.09136). Lays out a RAG taxonomy (Naive/Advanced/Modular/Graph/Agentic); since DeepDelve is already multi agent, any RAG added is Agentic RAG by this paper's own taxonomy too, reinforcing the finding above that this is the field's least validated category.
(arXiv:2510.03847, arXiv only, not confirmed peer reviewed). Its central finding: the primary bottleneck for small models is orchestration and I/O, not knowledge, and the biggest lever for tool use reliability is schema constrained decoding, not model size or RAG. Its own ablation: an 8B model with schema constrained decoding plus INT8 quantization beats an unconstrained larger baseline (98.7/97.9% valid output/execution versus 91.2/89.4%), while the same 8B model without the constraint drops to 94.3/90.8%. This reinforces an already-open DeepDelve candidate, forced tool_choice on vLLM, as a more direct fix for the actual observed tool use failures than adding RAG would be. Its recommended "SLM default, LLM fallback" architecture is close to DeepDelve's own specialist tiering (already tried and found a net loss on this hardware), the missing piece being structured decoding and validators, not the routing idea itself. A relayed claim that RAG gets a 1B model to 30B-equivalent performance is more modest and conditional in the actual literature than that magnitude suggests. Read cover to cover: its own limitations section notes benchmark/API drift risk and heavy validator dependence hiding reasoning failures, the constrained decoding win is real and measured, but not guaranteed to generalize past its own benchmark.
RAG isn't the most directly load bearing fix for what actually broke in live testing; both real MiniCPM failures match tool-selection/execution guardrail gaps the SLM survey attributes to missing schema constrained decoding, not a knowledge gap. If RAG is built, it should be scoped narrow (Agentic RAG by construction, the least validated category, a reason for caution, not avoidance).
A correction: the "prior RAG failure" turned out to be in this project's own history, a deleted exact-string-match Q&A cache with no embeddings or retrieval at all, and its real failure was a benchmark isolation bug (a later model's trial hit an earlier model's cached answer), not a cascade blindness case this taxonomy would predict. This narrows the actual justified goal: a persistent cross-run knowledge cache supplementing web search, not a full multi-hop Graph RAG layer, since DeepDelve's own Searcher-to-Analyzer chain already provides multi-hop reasoning without vector retrieval. Three graph RAG projects (LightRAG, RAG-Anything, microsoft/graphrag) got reviewed as possible complements and rejected as architecturally heavier than needed: LightRAG requires an LLM call per chunk for entity extraction and 4 storage backends; RAG-Anything targets multimodal content DeepDelve doesn't have a gap for; GraphRAG is built for global sensemaking over large static corpora, the wrong shape for caching individual verified facts across runs.
Prompted by the user noticing most public agent projects converge on SKILL.md style playbook files rather than a structural verification pipeline, and wanting to know if DeepDelve has drifted into genuinely undocumented territory or is reinventing something with a name. Researched via primary sources: repo READMEs, papers, Anthropic's own engineering writeup.
What already exists elsewhere, each with real prior art: deterministic verification over agent output exists but concentrated in domains with a natural pass/fail oracle (a ZenML case study on a Claude Code hook verifying against test criteria is the closest match; code has tests, open ended research doesn't). NLI based citation grounding is a real published technique (VeriCite, SIGIR-AP 2025). VERIMAP (EACL 2026, arXiv:2510.17109) is the closest academic analog to a structural check pipeline, task general rather than research specific, worth adopting its DAG-shaped "one verification function per subtask" framing to formalize DeepDelve's own problem routing. Independent-context critic beating same-context self critique is literature backed, not just intuition (Reflexion and Self-Refine, the standard pattern most frameworks default to, use same-context self critique, which current literature says remains fundamentally unreliable without external signals); DeepDelve's fresh context PeerReviewer sits on the literature-backed correct side. Small frozen classifiers replacing LLM judgment is precedented (RouteLLM style), though for model routing, not DeepDelve's exact use case, and only found via a secondary source. A dedicated citation verification stage is precedented in Anthropic's own multi-agent research system writeup, but it stays LLM based throughout with no deterministic fallback.
What was checked and found not to have this: GPT Researcher's quality control is breadth-over-depth with no dedicated critic; Stanford STORM's "verification" is perspective diversity during research, not post hoc checking; dzhng/deep-research has no verification layer beyond re-querying; Tongyi DeepResearch addresses reliability through training, not an inference time structural layer. CrewAI, AutoGen, CAMEL, and LangGraph were only checked via search snippets in this first pass, unconfirmed.
Claude Skills were confirmed to solve a different problem: procedural knowledge, not verification. Independent commentary frames Skills as having their own reliability problems (activation failure, execution failure especially at late-stage verification steps), directly confirming the user's original intuition that playbook documents betting on instruction-following don't hold for local models under load.
Not novel at the level of individual components (deterministic checks, independent critics, NLI grounding, classifier routing, and a separated citation stage all predate DeepDelve, each with a real citable source), but no source combines dozens of named priority-ordered checks plus a starvation guard plus a regression test over the verification logic itself, all applied to the local sub-30B regime specifically. Honest framing: "a disciplined, unusually deep combination of known techniques applied to an underserved regime," not "invented verification from scratch."
Same day, all 6 remaining frameworks checked via primary sources. CrewAI's function-based guardrails are the closest of the six to DeepDelve's structural checks, but failures retry in the same loop, no independent reviewer, no starvation guard, no regression test. AutoGen/AG2's critic shares the same conversation transcript, not a fresh context (closer to Reflexion's same-context category); one notebook that could show real deterministic tool-grounded feedback couldn't be fetched, left unconfirmed. CAMEL-AI's Critic agent is purely LLM based, and its own wiki concedes it "may not really solve the fundamental extrapolation problem." LangGraph's Corrective RAG example was the most interesting partial match: document relevance grading is an LLM call, but on failure it deterministically routes to a real web search tool instead of retrying the same LLM, the branch is deterministic even though the judgment producing it isn't; still no independent critic or structural check. Perplexity's engineering writeups show real depth but entirely about retrieval and ranking, no documented citation correctness mechanism either way. OpenAI Deep Research's system card, read directly, is a safety document, not an architecture writeup, citations mentioned once in passing with no described mechanism.
Updated verdict: every system examined converges on same-context LLM-as-judge critique (sometimes with one deterministic routing branch) or no documented mechanism at all. None combines an independently instantiated fresh-context reviewer with a priority-ordered bank of dozens of named checks and a regression test over the verification logic itself, and none targets the local sub-30B regime DeepDelve does. The framing stands: a disciplined combination of known techniques applied to an underserved regime, not an invention from scratch.
History
Model Research
Reviews & Audits
Reference