Point it at any Obsidian vault and have a grounded, cited conversation with your notes — from Claude Code, via MCP. ObsiRag treats the vault as a knowledge graph, not a bag of flat files: it parses Obsidian structure, chunks by section, retrieves with hybrid search + depth-1 graph traversal, and answers with citations back to the exact note and section.
This is a portfolio project built to demonstrate an understanding of RAG's real failure modes — bad chunking, retrieval misses, hallucination, multi-note questions — and how a design mitigates each, with an honest evaluation that measures whether the graph-aware differentiator actually works.
Local-first. Runs on your machine. Your vault never leaves it (local embeddings; only the final answer step calls an LLM). Config lives in
.env; nothing about your vault is committed.
Obsidian vault (*.md)
│
┌──────────────────────────────┴──────────────────────────────┐
│ Indexing (offline) │
│ parser ──▶ chunker ──▶ embedder (dense) ─┐ │
│ │ │ sparse (BM25) ─┐│ │
│ │ │ ▼▼ │
│ NoteGraph token-bounded Qdrant collection │
│ (wikilinks) section chunks (dense + sparse vectors) │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────────────────┴──────────────────────────────┐
│ Query (online) │
│ question ─▶ Retriever ─▶ Generator ─▶ grounded cited answer │
│ ├ hybrid search (dense + BM25, RRF fusion) │
│ └ depth-1 graph traversal (linked notes) │
└──────────────────────────────┬──────────────────────────────┘
│
MCP server (search_vault · query_vault) ──▶ Claude Code
| Module | Responsibility |
|---|---|
tokenizer.py |
Token counting aligned to the embedder's real window (single source of truth) |
parser.py |
Obsidian markdown → frontmatter, heading sections, normalized [[wikilinks]], tags |
chunker.py |
Section-aware, token-bounded chunks with contextual-header breadcrumbs |
embedder.py / sparse_embedder.py |
Local dense embeddings (BGE) / BM25 sparse (fastembed) |
vector_store.py |
Qdrant: dense + sparse vectors, hybrid RRF search, note-filtered fetch |
graph.py |
Wikilink resolution + depth-1 neighbours (outgoing + backlinks) |
retriever.py |
Hybrid search → primary chunks, then graph-enriched linked context |
generator.py |
Claude call with a grounded prompt; parses [n] citations |
evaluation.py |
Recall/precision + graph-uplift, faithfulness judge |
config.py / pipeline.py / mcp_server.py |
Env config, end-to-end wiring, MCP tool |
- Section-aware chunking, bounded by the embedder's real token window. Chunks follow heading sections, not fixed-size cuts, so each chunk is a coherent unit and citations point at a real section. The size ceiling is coupled to the embedding model's actual
max_seq_length— anything longer is silently truncated before embedding (no error, retrieval just degrades). A guardrail test asserts no chunk can exceed it. BAAI/bge-small-en-v1.5(512-token window) overall-MiniLM-L6-v2(256): the larger window fits section-sized chunks without fragmenting them, with stronger retrieval benchmarks — still tiny and fully local.- Contextual-header breadcrumbs. Flat per-heading chunks keep citations granular; prepending the ancestor heading path (
# Overview > ## Core Concepts) gives each chunk its parent context without the chunk-size blow-up of true hierarchical nesting. - Hybrid search fused with RRF. Dense (semantic) + BM25 (exact keyword) via Qdrant's Reciprocal Rank Fusion. RRF fuses on rank, sidestepping the scale mismatch between bounded cosine and unbounded BM25 that naive weighted-score fusion falls into.
- Depth-1 graph traversal — the differentiator. When a note is retrieved, its linked/backlinked notes' chunks are pulled in as supporting context. The vault is a graph, not flat files. The linked context is relevance-ranked against the query and limited to one chunk per note, so a highly connected hub note can't flood the context with off-topic neighbours (a failure mode found by running on a real 276-note, ~1,000-link vault).
- Cross-encoder reranking. Hybrid search + graph traversal are fast but rank passages independently; a local cross-encoder (
ms-marco-MiniLM) re-scores the top-20 candidates by reading each (query, passage) pair together, then keeps the top-k. On a real vault this pulled the canonical overview note from rank #8 into #1 — bi-encoder recall is cheap and wide, cross-encoder precision is expensive and narrow, so we use each where it pays. - Grounded generation with an injected client. The system prompt forbids outside knowledge and requires a citation per claim;
claude-opus-4-8is called with notemperature(the current model rejects sampling params) — factuality is steered by the prompt. The Anthropic client is injected, so the whole pipeline is testable without API calls.
| Failure mode | Mitigation | Verified by |
|---|---|---|
| Silent truncation (chunk > embedder window, dropped without error) | Chunk ceiling coupled to the model's real token window | Guardrail test: no chunk's token_count > max |
| Semantic miss (query wording ≠ note wording) | BM25 sparse vectors catch exact terms dense embeddings miss | Hybrid test: exact-keyword query ranks the right note #1 |
| Ranking miss (the right note is retrieved but ranked just outside top-k) | Cross-encoder reranking: fetch top-20 by hybrid, re-score each (query, passage) pair, keep top-k | On a real vault, the canonical overview note went #8 → #1 |
| Insufficient context (multi-note questions) | Depth-1 graph traversal assembles the note's neighbourhood | Eval graph-uplift (below) |
| Hallucination | Ground-only prompt + mandatory [n] citations; refusals handled |
Generator tests; citation extraction |
| Scale-mismatch fusion | RRF (rank-based) instead of weighted raw scores | Design; hybrid retrieval tests |
The eval runs over a purpose-built 16-note interlinked sample vault (eval/sample_vault/), not the tiny test fixture — because recall over a toy vault saturates at ~100% and proves nothing. Ground truth is note-level (a documented proxy for chunk-level relevance).
The headline metric is graph uplift: recall on the hybrid-search primary hits vs. recall once depth-1 graph traversal adds linked notes. A k-sweep (varying only the retrieval budget over the fixed, pre-committed eval set — every k published, no cherry-picking), run both without and with the cross-encoder reranker. A representative run:
BASELINE (hybrid + graph) WITH RERANKER
k recall(primary) recall(+graph) precision | recall(primary) precision
1 0.667 0.96–1.00 1.000 | 0.604 0.875
2 0.833 1.000 0.688 | 0.896 0.750
3 0.938 1.000 0.542 | 1.000 0.583
5 1.000 1.000 0.375 | 1.000 0.375
What it shows — two findings, both honest:
- Graph traversal recovers expected notes hybrid search misses —
recall(+graph)reaches 1.000 whilerecall(primary)is still 0.67–0.94 at low k, i.e. ≈+30% recall uplift at k=1, fading to 0 as the budget grows and hybrid saturates.graph_upliftis ≥ 0 by construction, so it can't be gamed. Graph-aware retrieval matters most under a tight retrieval budget (and on larger vaults). - Reranking is not a free lunch. It lifts both recall and precision in the practical k=2–3 range (at k=3, recall 0.938→1.000, precision 0.542→0.583) but slightly hurts k=1 on this tiny vault (the cross-encoder's single top pick occasionally disagrees with note-level ground truth, and k=1 has no room for error). At k=5 everything converges.
The sample vault understates the reranker. Its 16 short, distinct notes are easy for hybrid search, so reranking has little to fix. On a real 276-note vault the effect is dramatic — the canonical
PolyBlitz Overviewnote went from rank #8 → #1 once reranked (shown live above). Reranking earns its keep on large, noisy vaults, exactly where deep-ranked relevant notes hide.
On the ranges: Qdrant's HNSW index is approximate, so a re-index shuffles which chunks land at the top-k margins — exact decimals drift a little run-to-run; the pattern is stable. Don't read the third decimal as gospel; read the shape. Reproduce with
uv run python eval/run_eval.py.
Honest limits (the point of an eval is to know them):
- Recall@k is one proxy. Graph traversal's other purpose is context enrichment for the generator on multi-note questions — answer completeness, which this metric doesn't capture.
- Note-level ground truth is a documented proxy for chunk-level relevance; the sample vault is small (recall saturates by k=5).
- Faithfulness uses an LLM-as-judge with structured output; Opus judging Opus is self-evaluation bias — the number is directional, not ground truth. A human spot-check remains the gold standard. (The judge is built and unit-tested; wiring it across the eval set with a live client is future work.)
Prerequisites: Python 3.13, uv, Docker (for Qdrant). No Anthropic API key is needed for the Claude Code flow — Claude Code is the LLM.
Inside Claude Code, Claude Code is already the LLM: the MCP server just retrieves the relevant notes, and Claude Code writes the grounded, cited answer.
# 1. Install dependencies + start Qdrant
uv sync
docker compose up -d
# 2. Register the MCP server with Claude Code — point it at YOUR vault (no key)
claude mcp add obsirag \
--env VAULT_PATH=/path/to/your/obsidian/vault \
-- uv run --directory /path/to/obsiRag python -m obsirag.mcp_server
# 3. Reconnect Claude Code, run /mcp to confirm "obsirag" is connected, then ask:
# "use obsirag to tell me what my notes say about <topic>"Prefer a JSON MCP config? (e.g. Claude Desktop)
{
"mcpServers": {
"obsirag": {
"command": "uv",
"args": ["run", "--directory", "/path/to/obsiRag", "python", "-m", "obsirag.mcp_server"],
"env": { "VAULT_PATH": "/path/to/your/obsidian/vault" }
}
}
}The server exposes two tools, picked by whether a key is set:
| Tool | Needs key? | What it returns |
|---|---|---|
search_vault (always) |
❌ No | The retrieved notes/sections with their text, as grounded context — the host assistant writes the cited answer from it. |
query_vault (only if ANTHROPIC_API_KEY is set) |
✅ Yes | A finished, cited answer (the server calls Claude itself) — for non-LLM MCP clients. |
First launch indexes your vault (~20s for a few hundred notes); later launches reuse the index (no re-embedding). After editing many notes, refresh with
OBSIRAG_REINDEX=1in the server'senv(or delete the Qdrant collection).
# Evaluate retrieval quality over the built-in sample vault (no vault or key needed):
uv run python eval/run_eval.py
# Search YOUR vault from the terminal — retrieval only, no key:
uv run python -m obsirag.search "what are my notes about X?"Only needed if a non-LLM client should get a finished written answer (not required for Claude Code). Copy cp .env.example .env, set VAULT_PATH and ANTHROPIC_API_KEY, and the MCP server additionally exposes query_vault. The key powers only that final answer-writing step — parsing, embeddings, retrieval, and the eval are all local.
uv run pytest -q110+ tests. Logic runs offline against stubs (no API cost); Qdrant-backed and live-API tests skip cleanly when Qdrant is down or no key is set. Built test-first (TDD) throughout, with a two-stage review per increment.
src/obsirag/ parser, chunker, tokenizer, embedder, sparse_embedder,
vector_store, graph, retriever, generator, evaluation,
config, pipeline, mcp_server
eval/ sample_vault/ (16 notes), eval_set.json, run_eval.py
tests/ per-module tests + fixture vault
docs/ design spec + per-MVP implementation plans
Built incrementally as six MVPs (parse/chunk → embed/store → retrieve/graph/hybrid → generate/cite → evaluate → MCP), each a reviewed, self-contained slice. Design docs and plans live in docs/superpowers/.
Delta re-indexing (only re-embed changed notes) · faithfulness judged across the full eval set with a live client (distinguishing "judge failed" from "unfaithful") · render-time handling of unresolved citation markers · section-level ground truth · capping very high-degree hub notes in graph traversal · a bigger/messier eval vault where reranking's gains (visible on the real 276-note vault) are measurable.
MIT — see LICENSE. Do whatever you like with it; attribution appreciated.