Landscape Survey

Deep Research Agent Landscape: Architectural Innovations for DeepDelve

This document reviews the broader landscape of state-of-the-art open-source deep research agents and outlines advanced architectural patterns that can be adapted to improve DeepDelve.

---

1. The Deep Research Landscape

We analyzed four major open-source deep research systems:

1. GPT Researcher (assafelovic/gpt-researcher)
2. STORM (stanford-oval/storm)
3. Tongyi DeepResearch (Alibaba-NLP/DeepResearch)
4. Open Deep Research (langchain-ai/open_deep_research)

---

2. Deep-Dive: System Profiles & Architectural Wins

A. GPT Researcher (Parallel Synthesis & Sub-topic Outlining)

• Concept: Focuses on speed and comprehensive web scraping. It uses a coordinator agent to break down a main query into a set of distinct sub-topics (e.g. 5-10 sub-queries), dispatches scraping agents to run searches and fetch pages in parallel, and compiles them.
• Key Innovation: Parallel Crawling & Scraping. Rather than a single searcher doing sequential search-then-fetch operations, GPT Researcher spawns multiple scraping processes concurrently, caches raw text, and uses a dedicated summarizer tool to prune context before synthesis.
• Actionable for DeepDelve:
  • Concurrent Specialists: Refactor delegate_tasks to run WebSearcher tasks completely concurrently rather than blocking sequentially in asyncio.gather.
  • Context Pruning (Summarizer Tool): Introduce a token-pruning utility that filters out raw HTML boilerplate (navs, footers, scripts) before writing markdown files to the workspace, saving context window space for the Analyzer models.

---

B. Stanford STORM (Multi-Perspective Curation & Outline Synthesis)

• Concept: Focuses on pre-writing planning to avoid the "consensus bias" of standard RAG models. It simulates a panel of diverse personas (e.g., an academic, a critic, a builder) who interview each other and search for sources.
• Key Innovations:
  • Multi-Perspective Persona Generation: Before drafting a research plan, the system generates 3-5 distinct personas (e.g. DevOps Engineer vs. Database Admin when researching database extensions) and has them brainstorm questions from their respective domains.
  • Hierarchical Outline Pre-Synthesis: STORM synthesizes a stable outline (mind map) of sections before any actual text is drafted. Text is then written incrementally section-by-section.
• Actionable for DeepDelve:
  • Persona Brainstorming Phase: Add a pre-planning phase. The Planner uses think_tool to generate three relevant perspectives, brainstorms questions for each, and maps these directly to the _todos.md slots.
  • Sectional Drafting: Instead of the Planner writing the entire final_report.md in one massive LLM call (which often runs out of output tokens), the Planner should write the report section-by-section using the structured outline, appending sections sequentially.

---

C. Tongyi DeepResearch (Test-Time Search Scaling)

• Concept: Leverages test-time scale (test-time computing) via RL (GRPO) to run long-horizon search and reasoning loops. It uses "Heavy Mode" to drastically expand search queries.
• Key Innovation: Test-Time Search Scaling (Heavy Mode). When a query is complex, the model scales its inference compute by dynamically generating alternative queries and searching deeply into the search results page (fetching top 5-10 results rather than just the top 1).
• Actionable for DeepDelve:
  • Heavy Mode Config: Add a settings.search_mode: "light" | "heavy" parameter.
  • Query Expansion Engine: In "heavy" mode, when web_search is called, the agent automatically expands the query into 3 distinct search strings, executes them all, and auto-fetches the top result from each, expanding the verified data pool.

---

D. Open Deep Research (Human-in-the-Loop & MCP Tooling)

• Concept: Built on LangGraph, it features feedback loops for iterative query reformulation and incorporates Human-in-the-Loop (HITL) gates.
• Key Innovations:
  • Interactive Plan Approval: The agent pauses after generating its initial research plan, allowing the user to edit or approve the TODOs before any search quotas are consumed.
  • Model Context Protocol (MCP) Integration: Integrates external tools via MCP, allowing the agent to write research straight to local databases, Obsidian vaults, or query local documents.
• Actionable for DeepDelve:
  • Human-in-the-Loop (HITL) Gate: Add settings.human_in_the_loop: true. When enabled, the CLI/TUI pauses after write_todos and asks the user: "Review plan in _todos.md. Press Enter to approve, or edit the file and type 'r' to reload."
  • MCP Client Loader: Implement an MCP loader in src/tools/ to dynamically load third-party tools (like semantic search engines or paper lookup databases) at runtime.

---

3. High-Impact Refactoring Proposals for DeepDelve

To integrate these advanced patterns, we propose the following evolutionary steps for DeepDelve:

flowchart TD
    A[User Query] --> B[Phase 1: Multi-Perspective Persona Generation]
    B --> C[Phase 2: Bounded Outline Planning]
    C -->|HITL Gate: User Approval| D[Phase 3: Test-Time Search Scaling / Heavy Mode]
    D --> E[Phase 4: Concurrency-Optimized Scraping & Pruning]
    E --> F[Phase 5: Section-by-Section Drafting & Peer Critique]
    F --> G[Final Report]

Loading

1. The Persona Brainstorming Stage

Update the PLANNER_INSTRUCTIONS workflow to mandate a brainstorming step:

"Before calling write_todos, use think_tool to define 3 expert personas for the topic. Generate 2 crucial questions from each persona's perspective. Map these questions into your named planning slots."

2. Test-Time Search Scaling (Heavy Mode)

Implement query expansion inside tools/web.py when search_mode is set to "heavy":

• Input query: "Compare Elasticsearch and pgvector"
• Expanded queries:
  1. "Elasticsearch vs pgvector indexing algorithm performance"
  2. "pgvector scaling limitations in production"
  3. "Elasticsearch hybrid vector keyword search setup"
• Execute all three concurrently, auto-fetching the top-1 result of each.

3. Human-in-the-Loop (HITL) Planning Gate

Introduce a physical check in engine/tui.py:

if config.cfg.get("settings", {}).get("human_in_the_loop", False):
    # Wait for user keypress or TUI Button event before calling delegate_tasks
    await self.wait_for_user_plan_approval()

This prevents the agent from wasting LLM tokens and API quotas on plan paths that don't match the user's intent.