-
Notifications
You must be signed in to change notification settings - Fork 0
1.14 Obsidian Vault Guide
New in v1.0.0-rc.2. ClioDeck can point at an Obsidian vault folder, index the notes inside it, and treat them as a first-class RAG source alongside your PDFs (secondary) and Tropy archives (primary).
This integration was absorbed from the ClioBrain project during the RC2 fusion.
-
Indexing. ClioDeck walks the vault, parses markdown notes (frontmatter, wikilinks, tags), chunks them and stores embeddings in the shared
.cliodeck/brain.db(obsidian_*-prefixed tables — folded in from the older standaloneobsidian-vectors.dbfile, which no longer exists on a fresh project; see "Storage and migration" below). - Retrieval. Every Brainstorm turn pulls notes from the vault into its retrieval system prompt, with a citation back to the note path. The retrieval scope toggles in the chat panel let you mix biblio + primary + notes in any combination.
-
MCP exposure. The vault is queryable from external MCP clients via the
search_obsidiantool on ClioDeck's MCP server (BM25 / FTS5). -
Partial-success reporting. Indexing is never binary: the index report shows
indexed/skipped/failedcounts.
The vault is read-only from ClioDeck's point of view. Notes are not modified; nothing is written back to your .md files.
Settings → Obsidian Vault (VaultConfigSection):
- Click Link a vault….
- Pick the vault folder — that's the folder Obsidian itself opens (the one containing the
.obsidian/directory, not a subfolder). - The path appears in the status block.
- Click Index to start the first build.
The index is per-project. If you switch ClioDeck projects, the vault link doesn't follow — each project decides whether it wants a vault and which one.
-
Index — incremental: only notes that changed since the last run are reprocessed.
-
Reindex — forces a full rebuild. Use it after you change embedding model in Settings → LLM, or if you suspect drift.
-
Unlink — drops the link to the vault folder. Re-linking the same folder triggers a full rebuild from scratch, not a reuse of the previous index.
⚠️ Real bug, verified against code: Unlink also deletes the vault's index file at the filesystem level, and since the db consolidation that path is.cliodeck/brain.db— the file also holding your PDF vectors, Tropy primary sources and research journal. Unlinking therefore deleted the whole shared database. Fixed in RC4: only the vault's own tables are dropped. Since RC4 the action also asks for confirmation, and says that your Obsidian notes themselves are never touched.
While indexing runs, the section shows live progress (<stage> — <processed>/<total>).
Fixed in RC4. The indexing job always wrote to the correct project, but its progress events carried no project identifier: switching project mid-index and reopening Settings could show the old job's numbers against the new project. The progress event is now scoped.
When indexing finishes:
✓ Indexed: 412 · Skipped: 7 · Failed: 0
The Skipped count includes notes the parser intentionally left out (empty notes, frontmatter-only files, binary files masquerading as .md, etc.). Failed is for I/O or parse errors — investigate those.
In the Brainstorm chat panel, the source-type toggles control which retrievers run:
| Toggle combination | Effect |
|---|---|
| Biblio only | Secondary sources (PDFs) only. |
| Primary only | Tropy archives only. |
| Notes only | Vault only. |
| Biblio + Primary (no notes) | PDFs + Tropy. |
| Biblio + notes | PDFs + vault. |
| Primary + notes | Tropy + vault. |
| All three | PDFs + Tropy + vault (the default). |
Vault chunks come into the prompt with type : note and a TITRE field that's normally a short title — the note's frontmatter title, or its first H1 heading, or the bare filename without folder path or extension (extractTitle() in ObsidianMarkdownParser.ts). The full relative path (e.g. Reading notes/Hobsbawm-1962.md) is only used as a last-resort fallback when none of those exist (retrieval-service.ts: h.note.title || h.note.relativePath). Clicking a note citation in the assistant's reply opens it via the configured sources:open-note handler (typically the system's default markdown app — Obsidian itself, if it's registered).
If you've enabled ClioDeck's MCP server (see the MCP Integration Guide), the vault is reachable from Claude Desktop / Claude Code as the search_obsidian tool:
search_obsidian(query: string, topK?: int = 10)
The search is BM25 over FTS5, returning the top-K matching notes with a short snippet. It does not use dense embeddings — that's only for in-app Brainstorm retrieval.
The parser handles:
- Standard markdown body.
- YAML frontmatter (used for metadata; not embedded as content).
-
[[wikilinks]]and#tags(preserved in the text; useful as anchors). -
![[embeds]](the link itself is preserved; the embed is not followed).
The parser ignores:
- Files outside the vault root.
- The
.obsidian/directory. - Non-
.mdfiles (images, PDFs inside the vault, etc.). For PDFs you keep in your vault, add them to ClioDeck's bibliography to index them as secondary sources. - Notes skipped for a structural reason: hidden file, empty note, binary content, a frontmatter parse error, oversized, excluded by an ignore pattern, unreadable, or a note whose only content is broken wikilinks (
SkipReasoninbackend/integrations/obsidian/scan-report.ts). There is no user-settable frontmatter flag to opt a note out. There is also no per-note skip reason surfaced anywhere today — the indexer has no logging and the IPC handler only returns aggregate indexed/skipped/failed counts, not which note was skipped or why.
The vault index used to live in its own .cliodeck/obsidian-vectors.db file. A later consolidation (obsidianStorePath() in ObsidianVaultIndexer.ts) folded it into the same shared .cliodeck/brain.db that PDF and Tropy vectors already used, under obsidian_*-prefixed tables (obsidian_notes, obsidian_chunks, obsidian_chunks_fts) to avoid name collisions. Opening a project that still has the legacy standalone file triggers an automatic one-time migration (rename if brain.db doesn't exist yet, or ATTACH + INSERT OR IGNORE if it does) — you should never need to do this by hand. On project load, ClioDeck also auto-migrates older nested layouts (.cliodeck/v2/*) to the flat scheme via migrateWorkspaceToFlat, independently of this.
The index itself is purely derived data and safe to lose — but because it now lives in the shared brain.db, there is no longer a way to delete just the Obsidian index by removing a file; doing so removes your PDF and Tropy indexes and research-journal history too. Use Settings → Obsidian Vault → Reindex to rebuild the vault's own tables in place instead (see the Unlink warning above for the one UI path that still does a raw file delete).
⚠️ Residual risk, verified against code — downgraded after an adversarial re-check refuted this page's earlier, stronger claim. Four independent classes each open their ownbetter-sqlite3connection to the sharedbrain.db—VectorStore.ts(PDFs),PrimarySourcesVectorStore.ts(Tropy),HistoryManager.ts(research journal), and this page's ownObsidianVaultStore.ts— and only the Obsidian store setsjournal_mode = WAL. An earlier version of this callout claimed concurrent writes would fail immediately withSQLITE_BUSYbecause "better-sqlite3's default busy timeout is 0ms" — that was wrong: 0ms is raw SQLite's default, but better-sqlite3's constructor applies a 5000ms default timeout (database.js:'timeout' in options ? options.timeout : 5000), so every connection now waits up to 5 seconds for the lock, and all five writers ofbrain.dbset it — the manuscript store was still missing it until RC4. What survives is much weaker: a single write transaction holding the lock for more than 5 seconds, conceivable during a large batch index, could still push a concurrent writer intoSQLITE_BUSY.