# Obsidian Vault Guide

**New in v1.0.0-rc.2.** ClioDeck can point at an [Obsidian](https://obsidian.md) vault folder, index the notes inside it, and treat them as a first-class RAG source alongside your PDFs (secondary) and Tropy archives (primary).

This integration was absorbed from the [ClioBrain](https://github.com/inactinique/cliobrain) project during the RC2 fusion.

---

## What you get

- **Indexing.** ClioDeck walks the vault, parses markdown notes (frontmatter, wikilinks, tags), chunks them and stores embeddings in the shared `.cliodeck/brain.db` (`obsidian_*`-prefixed tables — folded in from the older standalone `obsidian-vectors.db` file, which no longer exists on a fresh project; see "Storage and migration" below).
- **Retrieval.** Every Brainstorm turn pulls notes from the vault into its retrieval system prompt, with a citation back to the note path. The retrieval scope toggles in the chat panel let you mix biblio + primary + notes in any combination.
- **MCP exposure.** The vault is queryable from external MCP clients via the `search_obsidian` tool on ClioDeck's MCP server (BM25 / FTS5).
- **Partial-success reporting.** Indexing is never binary: the index report shows `indexed` / `skipped` / `failed` counts.

The vault is **read-only** from ClioDeck's point of view. Notes are not modified; nothing is written back to your `.md` files.

---

## Linking a vault

**Settings → Obsidian Vault** (`VaultConfigSection`):

1. Click **Link a vault…**.
2. Pick the **vault folder** — that's the folder Obsidian itself opens (the one containing the `.obsidian/` directory, not a subfolder).
3. The path appears in the status block.
4. Click **Index** to start the first build.

The index is per-project. If you switch ClioDeck projects, the vault link doesn't follow — each project decides whether it wants a vault and which one.

---

## Indexing

- **Index** — incremental: only notes that changed since the last run are reprocessed.
- **Reindex** — forces a full rebuild. Use it after you change embedding model in **Settings → LLM**, or if you suspect drift.
- **Unlink** — drops the link to the vault folder. Re-linking the same folder triggers a full rebuild from scratch, not a reuse of the previous index.

  > **⚠️ Real bug, verified against code:** Unlink also deletes the vault's
  > index file at the filesystem level, and since the
  > [db consolidation](#storage-and-migration) that path **is**
  > `.cliodeck/brain.db` — the file also holding your PDF vectors, Tropy
  > primary sources and research journal. Unlinking therefore deleted the
  > whole shared database. **Fixed in RC4**: only the vault's own tables are
  > dropped. Since RC4 the action also asks for confirmation, and says that
  > your Obsidian notes themselves are never touched.

While indexing runs, the section shows live progress (`<stage> — <processed>/<total>`).

> **Fixed in RC4.** The indexing job always wrote to the correct project,
> but its progress events carried no project identifier: switching project
> mid-index and reopening Settings could show the old job's numbers against
> the new project. The progress event is now scoped.

When indexing finishes:

```text
✓ Indexed: 412 · Skipped: 7 · Failed: 0
```

The **Skipped** count includes notes the parser intentionally left out (empty notes, frontmatter-only files, binary files masquerading as `.md`, etc.). **Failed** is for I/O or parse errors — investigate those.

---

## Using the vault in Brainstorm

In the Brainstorm chat panel, the source-type toggles control which retrievers run:

| Toggle combination | Effect |
|---|---|
| Biblio only | Secondary sources (PDFs) only. |
| Primary only | Tropy archives only. |
| Notes only | Vault only. |
| Biblio + Primary (no notes) | PDFs + Tropy. |
| Biblio + notes | PDFs + vault. |
| Primary + notes | Tropy + vault. |
| All three | PDFs + Tropy + vault (the default). |

Vault chunks come into the prompt with `type : note` and a TITRE field that's normally a **short title** — the note's frontmatter `title`, or its first H1 heading, or the bare filename without folder path or extension (`extractTitle()` in `ObsidianMarkdownParser.ts`). The full relative path (e.g. `Reading notes/Hobsbawm-1962.md`) is only used as a last-resort fallback when none of those exist (`retrieval-service.ts`: `h.note.title || h.note.relativePath`). Clicking a note citation in the assistant's reply opens it via the configured `sources:open-note` handler (typically the system's default markdown app — Obsidian itself, if it's registered).

---

## Using the vault from external MCP clients

If you've enabled ClioDeck's MCP server (see the [MCP Integration Guide](./1.12-MCP-Integration-Guide.md)), the vault is reachable from Claude Desktop / Claude Code as the `search_obsidian` tool:

```text
search_obsidian(query: string, topK?: int = 10)
```

The search is BM25 over FTS5, returning the top-K matching notes with a short snippet. It does **not** use dense embeddings — that's only for in-app Brainstorm retrieval.

---

## What's indexed, what isn't

The parser handles:

- Standard markdown body.
- YAML frontmatter (used for metadata; not embedded as content).
- `[[wikilinks]]` and `#tags` (preserved in the text; useful as anchors).
- `![[embeds]]` (the link itself is preserved; the embed is not followed).

The parser ignores:

- Files outside the vault root.
- The `.obsidian/` directory.
- Non-`.md` files (images, PDFs inside the vault, etc.). For PDFs you keep in your vault, add them to ClioDeck's bibliography to index them as secondary sources.
- Notes skipped for a structural reason: hidden file, empty note, binary content, a frontmatter parse error, oversized, excluded by an ignore pattern, unreadable, or a note whose only content is broken wikilinks (`SkipReason` in `backend/integrations/obsidian/scan-report.ts`). There is no user-settable frontmatter flag to opt a note out. There is also no per-note skip reason surfaced anywhere today — the indexer has no logging and the IPC handler only returns aggregate indexed/skipped/failed counts, not which note was skipped or why.

---

## Storage and migration

The vault index used to live in its own `.cliodeck/obsidian-vectors.db` file. A later consolidation (`obsidianStorePath()` in `ObsidianVaultIndexer.ts`) folded it into the same shared `.cliodeck/brain.db` that PDF and Tropy vectors already used, under `obsidian_*`-prefixed tables (`obsidian_notes`, `obsidian_chunks`, `obsidian_chunks_fts`) to avoid name collisions. Opening a project that still has the legacy standalone file triggers an automatic one-time migration (rename if `brain.db` doesn't exist yet, or `ATTACH` + `INSERT OR IGNORE` if it does) — you should never need to do this by hand. On project load, ClioDeck also auto-migrates older nested layouts (`.cliodeck/v2/*`) to the flat scheme via `migrateWorkspaceToFlat`, independently of this.

The index itself is purely derived data and safe to lose — but because it now lives in the shared `brain.db`, there is no longer a way to delete *just* the Obsidian index by removing a file; doing so removes your PDF and Tropy indexes and research-journal history too. Use **Settings → Obsidian Vault → Reindex** to rebuild the vault's own tables in place instead (see the Unlink warning above for the one UI path that still does a raw file delete).

> **⚠️ Residual risk, verified against code — downgraded after an
> adversarial re-check refuted this page's earlier, stronger claim.** Four
> independent classes each open their own `better-sqlite3` connection to
> the shared `brain.db` — `VectorStore.ts` (PDFs),
> `PrimarySourcesVectorStore.ts` (Tropy), `HistoryManager.ts` (research
> journal), and this page's own `ObsidianVaultStore.ts` — and only the
> Obsidian store sets `journal_mode = WAL`. An earlier version of this
> callout claimed concurrent writes would fail immediately with
> `SQLITE_BUSY` because "better-sqlite3's default busy timeout is 0ms" —
> that was wrong: 0ms is raw SQLite's default, but better-sqlite3's
> constructor applies a **5000ms** default timeout
> (`database.js`: `'timeout' in options ? options.timeout : 5000`), so
> every connection now waits up to 5 seconds for the lock, and all five
> writers of `brain.db` set it — the manuscript store was still missing it
> until RC4. What survives is much weaker: a single write transaction
> holding the lock for **more than 5 seconds**, conceivable during a large
> batch index, could still push a concurrent writer into `SQLITE_BUSY`.

---

## See also

- [Brainstorm Mode Guide](./1.11-Brainstorm-Mode-Guide.md)
- [MCP Integration Guide](./1.12-MCP-Integration-Guide.md)