Idea: Semantic memory search & retrieval via MemoryIndexMiddleware

[Amine SBOUI (aminesbdev)]

Idea: Semantic memory search & retrieval via MemoryIndexMiddleware

Summary

Today, StoreBackend enables persistent cross-session memory in DeepAgents, but discovery is restricted to exact path lookups (read_file, ls) or literal substring matching (grep). As agents accumulate notes, preferences, and long-horizon facts across dozens of sessions, string-matching degrades rapidly.

We propose adding a MemoryIndexMiddleware that seamlessly hooks into persistent store operations (e.g. /memories/*) to maintain an embedding-backed semantic index, exposing a zero-overhead search_memory tool to the agent.

---

The Problem

1. Long-horizon memory lookup is brittle: When an agent writes memories across sessions (e.g., user preferences, debugging insights, project context), retrieving them later requires either:
   • The agent guessing the exact file path or naming scheme.
   • Using grep, which fails when phrasing differs (e.g., searching for "database configuration" when the file contains "PostgreSQL connection string").
2. Context Window Waste: Without semantic retrieval, agents often list directories (ls /memories/) and sequentially read multiple files (read_file), unnecessarily exhausting token budgets and latency.
3. No changes to BackendProtocol required: Storing and searching memory are separate concerns. Introducing semantic search shouldn't require bloating StoreBackend or altering BackendProtocol.

---

Proposed Solution: MemoryIndexMiddleware

By leveraging DeepAgents' extensible Middleware architecture, semantic indexing can be plugged in without touching core backend protocols or altering existing storage adapters:

[Agent Tool Call] 
       │
       ├── write_file("/memories/...") ──► [StoreBackend] ──► Persistent Store
       │                                         │
       │                               (on write/edit hook)
       ▼                                         ▼
[MemoryIndexMiddleware] ◄──────────────── Updates vector index / embeddings
       │
       └── Exposes tool: `search_memory(query: str, top_k: int = 5)`

Key Highlights:

• Pluggable Embeddings: Accepts any LangChain Embeddings model (OpenAIEmbeddings, HuggingFaceEmbeddings, local models via Ollama/vLLM).
• Lightweight Default In-Memory / Hybrid Vector Store: Works out of the box with zero external infrastructure (e.g., in-memory FAISS, LangGraph Store vector search if enabled, or simple cosine similarity over cached vectors).
• Route Filtering: Only indexes designated memory routes (e.g., matching /memories/** in a CompositeBackend setup), leaving workspace files untouched.
• Zero-Friction Integration: Compatible with existing create_deep_agent configurations.

---

Proposed Usage & API

from deepagents import create_deep_agent
from deepagents.backends import CompositeBackend, StateBackend, StoreBackend
from deepagents.middleware import MemoryIndexMiddleware
from langchain_openai import OpenAIEmbeddings
from langgraph.store.memory import InMemoryStore

store = InMemoryStore()
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Semantic memory middleware scoping /memories/ route
memory_indexer = MemoryIndexMiddleware(
    embeddings=embeddings,
    route_prefix="/memories/",
    top_k=5,
)

agent = create_deep_agent(
    model="openai:gpt-4o",
    backend=CompositeBackend(
        default=StateBackend(),
        routes={"/memories/": StoreBackend(store=store, namespace=lambda rt: (rt.server_info.user.identity,))},
    ),
    middleware=[memory_indexer],
)

Agent Tool Exposed:

@tool
def search_memory(query: str, top_k: int = 5) -> list[MemorySearchResult]:
    """Search persistent memory semantically for relevant past facts, preferences, or notes.
    
    Args:
        query: Natural language search description (e.g. 'user formatting preferences')
        top_k: Number of relevant snippets/files to return
    """

---

Why this belongs in DeepAgents

1. Fits the Philosophy: DeepAgents is the "batteries-included agent harness". Long-horizon context management and persistent memory are core pillars. Semantic recall is the natural missing piece between persistence (StoreBackend) and context efficiency.
2. Non-Breaking & Modular: Because it operates as middleware, it requires no breaking changes to StoreBackend, CompositeBackend, or BackendProtocol.
3. Synergy with LangGraph & LangChain: LangGraph's store already supports indexing/vector search capabilities in some backends; this bridges that capability directly into the DeepAgents tool suite.

---

Next Steps

I am happy to submit a clean Draft PR implementing:

• MemoryIndexMiddleware in libs/deepagents/deepagents/middleware/memory_index.py
• Unit tests validating index updates on write_file/edit_file and tool query returns
• An example notebook/script demonstrating persistent recall across distinct runs.

Would the maintainers be open to a PR for this? We'd love feedback on design and API ergonomics!

[sanjaykumar-xe]

This is an interesting direction, especially for long-running agents where manually navigating memory paths can become difficult.

One thing I’d be curious about is how the index would stay consistent when memories are updated or deleted. Would MemoryIndexMiddleware update the embedding index synchronously on each write, or would indexing happen asynchronously?

It might also be useful to have a fallback to the existing path-based lookup when semantic search returns low-confidence results, rather than making semantic retrieval the only path.

The namespace isolation in the proposed API also looks important, especially for multi-user deployments.

I’d definitely be interested in seeing a small benchmark comparing exact/path retrieval vs semantic retrieval for long-running agents.