Proposal: Separate organization, indexing, and retrieval for low-latency querying

[2Obe]

When using obsidian-find — both with and without semantic search — I noticed that retrieval can feel relatively slow. In my case, a query can take 20+ seconds end-to-end when executed through Codex, because retrieval is embedded in an agentic workflow that involves model reasoning, tool calls, and result interpretation.

For occasional searches this is fine. However, if the Second Brain is intended to become a real-time knowledge source — for example as context for another agent, application, or chat interface — retrieval should ideally happen in a few seconds or less.

This made me wonder whether knowledge organization and knowledge retrieval should become explicitly separated layers of the architecture.

1. Organization layer

Responsible for maintaining the canonical Second Brain:

• obsidian-ingest
• obsidian-reconcile
• obsidian-nightly
• synthesis and restructuring
• metadata and link maintenance
• other operations that improve the quality of the vault

These operations may be relatively expensive and can use a capable LLM. They do not necessarily need to be fast because they happen asynchronously or explicitly in the background.

2. Representation / indexing layer

Transforms the organized Markdown vault into representations optimized for retrieval.

For example:

• semantic embeddings
• lexical indexes
• metadata indexes
• potentially other derived representations in the future

This layer would be updated incrementally whenever notes change. The existing semantic index already seems to move in this direction.

The important distinction would be that the Markdown vault remains the canonical knowledge state, while indexes and embeddings are derived, disposable representations optimized for access.

3. Retrieval layer

Provides a fast, read-only interface to query the Second Brain.

Ideally, this layer would:

• avoid an agentic reasoning loop just to find relevant notes
• return structured results directly
• support lexical, semantic, or hybrid retrieval
• be independent of the LLM/harness used by the caller
• expose the same retrieval functionality through MCP, CLI, API, etc.

The caller could then decide what to do with the retrieved context. Codex, Claude, a small local model, or another application could all query the same Second Brain without requiring the retrieval implementation itself to depend on them.

Conceptually:

              ┌──────────────────────┐
              │  Organization Layer  │
              │                      │
              │ ingest / reconcile   │
              │ synthesize / nightly │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Canonical Markdown   │
              │        Vault         │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │ Representation Layer │
              │                      │
              │ lexical index        │
              │ embeddings           │
              │ metadata index       │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │   Retrieval Layer    │
              │                      │
              │ fast / read-only     │
              │ model-agnostic       │
              └──────────┬───────────┘
                         │
                ┌────────┼────────┐
                ▼        ▼        ▼
              Codex    Claude   Other apps

Why I think this could be useful

• Much faster interactive retrieval
• Clear separation between knowledge maintenance and knowledge consumption
• Embeddings become a derived representation rather than part of the knowledge itself
• Different layers can use different models or no LLM at all
• Retrieval becomes reusable by other applications and agents
• The Second Brain itself stays model- and harness-agnostic

There already seem to be several building blocks for this in the repository — particularly vault_ops.search, the MCP server, the semantic index, and obsidian-reindex.

So perhaps the main question is not whether these components exist, but whether it would make sense to make this separation an explicit architectural principle and provide a dedicated low-latency retrieval interface that bypasses the normal obsidian-find agent workflow.

What do you think? Would such a separation fit the direction of the project?

[eugeniughelbur]

This fits the direction, and the layer names match how the code already divides up. Thanks for writing it out with the pieces named.

Where I think you are right: the markdown vault is the canonical state and every index is derived and disposable. That is already true in practice, and /obsidian-reindex exists because a derived index has to be rebuildable from the notes alone. Saying it out loud as an architectural principle costs nothing and settles a class of future arguments.

On the 20 seconds, I want to separate two numbers before anyone optimizes. vault_ops.search() is ordinary Python and returns in well under a second on my vault. The rest of your 20 is the agent: reading the command file, deciding a query, calling a tool, reading results, deciding whether to search again, writing a reply. So the fast path you want is not a faster search. It is a way to call the search without the loop around it.

That path half exists. The MCP server already exposes obsidian_search as read-only and structured, and it does not care which model or CLI calls it, which is most of your retrieval layer. What it lacks is a caller that is not an agent.

The concrete gap is smaller than the proposal: vault_ops.search() has no CLI entry point. No python -m that takes a query and prints JSON. So a shell script, a cron job, or a non-MCP application cannot reach the index at all, and anyone who wants results today has to go through a model. That is one file, and it would give you the sub-second number without changing the architecture.

Two things I am not ready to commit to. A separate long-running retrieval service, because it turns a plain-files project into something with a process to babysit. And a stable retrieval API contract, because the index shape is still moving and I want to break it a few more times before promising it.

One question so I size this right: when you say retrieval should feed another agent or application, can the caller you have in mind speak MCP? If yes, the work is to document obsidian_search as the fast path and measure it. If no, the CLI entry point is the thing to build first, and I would sooner build that than a layer.

[2Obe]

Thanks, this clarification helps a lot. I agree that I was probably mixing two different issues: search performance itself and the agentic workflow around the search.

If vault_ops.search() already returns in well under a second, then the latter is really the problem I am trying to solve. In that sense, I also agree that this does not require a separate retrieval service or another architectural layer implemented as a running process.

What I had in mind with the “retrieval layer” is more of a logical interface boundary: organization can remain LLM-driven and relatively expensive, while consuming the resulting knowledge should be possible through a deterministic, low-latency path without requiring another reasoning loop.

Regarding your question about MCP: for Codex and similar clients, MCP is absolutely sufficient and obsidian_search already seems to provide exactly that fast path. My broader use case, however, also includes applications or lightweight agents that may not speak MCP. For those, a thin CLI around vault_ops.search() that returns structured JSON sounds like the right first step.

So I would now see the practical progression as:

1. document and benchmark obsidian_search as the fast retrieval path for MCP-capable clients;
2. add a minimal CLI entry point around the same underlying search for non-MCP callers;
3. keep both as thin interfaces over the existing search implementation rather than introducing a separate service.

Beyond that minimal path, I see two additional avenues that might be worth exploring independently:

Automatic re-indexing.
If the index is a derived representation of the Markdown vault, it would be useful if it could stay synchronized automatically. Rather than manually triggering obsidian-reindex, changed, added, or deleted notes could incrementally update the corresponding index entries. This would make the retrieval path much more suitable as a continuously available knowledge source without changing the canonical role of the Markdown files.

Alternative retrieval/index backends.
The same separation could also make it possible to experiment with additional representations or backends such as ChromaDB, LanceDB, or similar vector stores. I would not necessarily replace the existing implementation, but treat these as optional derived indexes behind the same retrieval interface. That could make it possible to compare latency, incremental updates, filtering, hybrid retrieval, and scaling behavior without coupling the vault itself to a particular database.

So the broader picture I have in mind is not necessarily a new service, but a clean boundary:

Markdown vault → derived/indexed representations → thin retrieval interfaces → consumers

with automatic synchronization and alternative index implementations being possible extensions rather than prerequisites.

And I still think making one architectural principle explicit could be valuable: the Markdown vault is the canonical knowledge state; indexes are derived representations; retrieval interfaces are consumers of those representations.

That distinction seems useful even if the implementation remains much smaller than the “three layers” in my original proposal.

[2Obe]

Let me add an final thought. IMHO one important design choice to make is to agree on the scope of the obsidian second brain.
Is it wanted to provide a complete "Retrieval Layer" with all the beneficial features like automatic re-indexing, identification of stale notes, selection of multiple vector-db providers? Or is the scope rather to "provide the organized obsidian knowledge base" --> Another tool, LLM, Harness, ... just uses it as markdown files? E.g. you plug it into OpenWebUI or AnythingLLM as Knowledge-Source and the indexing and re-indexing etc. is covered by these.