feat(obsidian): headless CLI vault ingestion (hindsight-obsidian-sync)

[benfrank241]

Closes #3128.

What & why

The Obsidian integration only runs inside the desktop app (isDesktopOnly: true), so a vault hosted on a headless server (kept current on disk by Obsidian Sync) can't be ingested without either running the full Electron app hidden or hand-rolling ingestion against the HTTP API and losing the plugin's carefully-designed behavior. @0xble asked for a supported headless path.

The codebase was already set up for this: SyncEngine is written against the minimal SyncVault/SyncFile interfaces and dependency-injects the vault, sync index, and persist writer — the only Obsidian coupling was requestUrl in the client. So this is a thin second frontend over the same engine, which means a server-synced vault produces identical document ids, scope tags, and prune-ownership as the plugin — the two ingesters never fight or duplicate.

How it's built (3 commits)

1. refactor: transport seam. Introduce a Transport abstraction so HindsightClient no longer imports obsidian. The plugin injects an obsidianTransport (requestUrl, to escape the renderer CORS sandbox); the CLI injects a fetch transport. One client, one set of request semantics — no divergent copies. Existing client tests pass unchanged (they switch from mocking requestUrl to injecting a fake transport).
2. feat: the Node frontend (src/node/):
   • fs-vault.ts — filesystem SyncVault (recursive *.md, POSIX-relative paths, ms mtime/ctime, skips dotfolders)
   • fetch-transport.ts — fetch-based Transport
   • json-index.ts — sync index persisted to JSON, atomic write; defaults to ~/.hindsight/obsidian/<vault>.json (outside the vault, so Obsidian Sync never propagates it)
   • cli.ts / cli-bin.ts — hindsight-obsidian-sync reconcile --vault <p> --bank <id> with env fallbacks, --include/--exclude/--prefix-doc-id, and a chokidar --watch mode
   • Packaging: second esbuild target builds dist/cli.js (node, shebang); package.json gains the bin, a files allowlist, and chokidar. The plugin build (main.js) and the release mirror are untouched.
3. test: real-framework + e2e coverage (see below).

Usage

npm install -g @vectorize-io/hindsight-obsidian
hindsight-obsidian-sync reconcile --vault ~/Vaults/Brain --bank my-vault \
  --api-url https://api.hindsight.vectorize.io --api-token hsk_...
# or live: add --watch

Testing — 90 tests (was 46)

Deterministic units plus real-framework and end-to-end suites:

• Real chokidar watch (watch.spec.ts) — drives an actual watcher over a temp vault: create → ingest, modify → re-ingest, delete → prune, non-markdown ignored.
• Real HTTP e2e (e2e-http.spec.ts) — runCli against a live node:http server, nothing mocked: asserts the retain POST (bearer token, document id, scope tags), a real DELETE on prune, and exit-1 when unreachable.
• Full-stack reconcile over a real temp vault: create/skip/update, mtime-only touch, delete, rename, include/exclude scoping, prefixDocId, prune-ownership, frontmatter tags + created-date timestamp, empty-body skip.
• Dual-ingester parity — the filesystem and in-memory (plugin) vaults emit byte-identical retain requests.
• Transport — client error propagation, health(), fetch rejection, cross-transport request parity.

tsc --noEmit clean; both bundles build. The existing test-obsidian-integration CI job (install → tsc → build → test) covers all of it.

Notes

• ~/.hindsight/obsidian/<vault>.json index location is deliberately out-of-vault; the README documents the shared-scope constraint when running the CLI and plugin against the same bank.
• @vectorize-io/hindsight-obsidian is already npm-published on release (the release workflow runs npm run build — which now also emits dist/cli.js — and npm ci before publish), so npm install -g gets a working hindsight-obsidian-sync bin with chokidar.

Thanks to @0xble for the well-scoped request and the code reading that made the direction obvious.