Landscape Survey

Deep Research Agent Landscape: Architectural Innovations for DeepDelve

This document reviews the broader landscape of state of the art open source deep research agents and lays out advanced architectural patterns that could be adapted to improve DeepDelve.

The deep research landscape

We looked at four major open source deep research systems: GPT Researcher (assafelovic/gpt-researcher), STORM (stanford-oval/storm), Tongyi DeepResearch (Alibaba-NLP/DeepResearch), and Open Deep Research (langchain-ai/open_deep_research).

System profiles and architectural wins

GPT Researcher: parallel synthesis and subtopic outlining. GPT Researcher focuses on speed and comprehensive web scraping. A coordinator agent breaks a main query down into a set of distinct subtopics (typically 5 to 10 sub queries), dispatches scraping agents to run searches and fetch pages in parallel, and then compiles everything together. Its key innovation is parallel crawling and scraping: rather than a single searcher doing sequential search then fetch operations, GPT Researcher spawns multiple scraping processes at once, caches raw text, and uses a dedicated summarizer tool to prune context before synthesis. Two things here are actionable for DeepDelve. First, concurrent specialists: we could refactor delegate_tasks so WebSearcher tasks run fully concurrently instead of blocking sequentially in asyncio.gather. Second, context pruning: a token pruning utility could filter out raw HTML boilerplate, navs, footers, scripts, before writing Markdown files to the workspace, which would save context window space for the Analyzer models.

Stanford STORM: multi perspective curation and outline synthesis. STORM focuses on prewriting planning to avoid the consensus bias that standard RAG models tend to have. It simulates a panel of diverse personas, say an academic, a critic, and a builder, who interview each other and search for sources together. Its two key innovations are multi perspective persona generation, where the system generates 3 to 5 distinct personas before drafting a research plan (for example a DevOps Engineer versus a Database Admin when researching database extensions) and has them brainstorm questions from their own domains, and hierarchical outline pre synthesis, where STORM builds a stable outline, a mind map of sections, before any actual text gets drafted, then writes the text incrementally section by section. For DeepDelve, this suggests a persona brainstorming phase: a pre planning step where the Planner uses think_tool to generate three relevant perspectives, brainstorms questions for each, and maps them directly onto the _todos.md slots. It also suggests sectional drafting: instead of the Planner writing all of final_report.md in one massive LLM call, which often runs out of output tokens, it could write the report section by section against a structured outline, appending sections as it goes.

Tongyi DeepResearch: test time search scaling. Tongyi DeepResearch leans on test time scaling via reinforcement learning (GRPO) to run long horizon search and reasoning loops, using a "Heavy Mode" to drastically expand its search queries. Its key innovation, test time search scaling, kicks in when a query is complex: the model scales its inference compute by dynamically generating alternative queries and searching deeper into the results (fetching the top 5 to 10 results rather than just the top 1). For DeepDelve, this points at a heavy mode config, a settings.search_mode: "light" | "heavy" parameter, paired with a query expansion engine: in "heavy" mode, when web_search is called, the agent would automatically expand the query into three distinct search strings, run all three, and auto fetch the top result from each, widening the verified data pool.

Open Deep Research: human in the loop and MCP tooling. Built on LangGraph, Open Deep Research features feedback loops for iterative query reformulation and builds in human in the loop (HITL) gates. Its two key innovations are interactive plan approval, where the agent pauses after generating its initial research plan so the user can edit or approve the TODOs before any search quota gets consumed, and Model Context Protocol (MCP) integration, which lets the agent write research straight to local databases, Obsidian vaults, or query local documents through external tools. For DeepDelve, this suggests a human in the loop gate: a settings.human_in_the_loop: true option that, when enabled, pauses the CLI or TUI after write_todos and asks the user something like "Review plan in _todos.md. Press Enter to approve, or edit the file and type 'r' to reload." It also suggests an MCP client loader in src/tools/ that could dynamically load third party tools, like semantic search engines or paper lookup databases, at runtime.

High impact refactoring proposals for DeepDelve

To bring these patterns together, here's a proposed evolutionary path for DeepDelve:

flowchart TD
    A[User Query] --> B[Phase 1: Multi-Perspective Persona Generation]
    B --> C[Phase 2: Bounded Outline Planning]
    C -->|HITL Gate: User Approval| D[Phase 3: Test-Time Search Scaling / Heavy Mode]
    D --> E[Phase 4: Concurrency-Optimized Scraping & Pruning]
    E --> F[Phase 5: Section-by-Section Drafting & Peer Critique]
    F --> G[Final Report]

Loading

The persona brainstorming stage. Update the PLANNER_INSTRUCTIONS workflow to require a brainstorming step: "Before calling write_todos, use think_tool to define 3 expert personas for the topic. Generate 2 crucial questions from each persona's perspective. Map these questions into your named planning slots."

Test time search scaling, heavy mode. Implement query expansion inside tools/web.py when search_mode is set to "heavy". Given an input query like "Compare Elasticsearch and pgvector," it would expand into something like "Elasticsearch vs pgvector indexing algorithm performance," "pgvector scaling limitations in production," and "Elasticsearch hybrid vector keyword search setup," all three run concurrently, auto fetching the top result of each.

The human in the loop planning gate. Introduce an actual check in engine/tui.py:

if config.cfg.get("settings", {}).get("human_in_the_loop", False):
    # Wait for user keypress or TUI Button event before calling delegate_tasks
    await self.wait_for_user_plan_approval()

This stops the agent from spending LLM tokens and API quota on plan paths that don't match what the user actually wanted.