feat(mcp): ephemeral-primary + persistent-attached two-database model

hyperdb-mcp/CHANGELOG.md

@@ -31,6 +31,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
   attempts are rate-limited to 3 per 60 seconds; exceeding the limit
   triggers daemon shutdown so the user sees the failure clearly
   rather than spinning silently.
+- **Two-database engine model.** Every session now has both an
+  ephemeral primary (created fresh per-session, deleted on exit) AND
+  a persistent attachment under the alias `"persistent"`. Unqualified
+  SQL routes to the ephemeral primary (the LLM's scratch space);
+  fully-qualified SQL like
+  `INSERT INTO "persistent"."public"."customers" ...` writes to
+  storage that survives across sessions.
+- **Platform-default persistent path.** When `--persistent-db` is
+  unset, the persistent file lives at the platform data directory:
+  `~/Library/Application Support/hyperdb/workspace.hyper` on macOS,
+  `~/.local/share/hyperdb/workspace.hyper` on Linux,
+  `%APPDATA%\hyperdb\workspace.hyper` on Windows. Override with
+  `HYPERDB_PERSISTENT_DB`.
+- **New CLI flags:** `--persistent-db <PATH>` (replaces `--workspace`,
+  which is kept as a deprecated alias with a stderr warning); and
+  `--ephemeral-only` to skip the persistent attachment entirely.
+- The `_table_catalog` and `_hyperdb_saved_queries` meta-tables now
+  live in the persistent attachment instead of the connection's
+  primary, so saved queries automatically persist across sessions
+  without any flag toggling.
+
+### Removed
+
+- **`--bare` flag.** Catalog seeding is now uniform: created when MCP
+  creates a fresh `.hyper`, never touched on existing files. Users
+  who want a pristine `.hyper` for export can `DROP TABLE _table_catalog`
+  once after creation; subsequent opens won't recreate it.
+
+### Performance
+
+- **Per-engine `_table_catalog` presence cache.** Catalog reads/writes
+  used to round-trip a `pg_catalog.pg_tables` probe on every call;
+  now the existence check is cached for the engine's lifetime and
+  primed immediately after `CREATE TABLE IF NOT EXISTS`.
+
+### Fixed
+
+- **Watcher recovery after hyperd restart.** The watcher's connection
+  pool now auto-rebuilds when a per-file ingest hits a connection-lost
+  error (typically after the daemon restarts hyperd). Each ingest gets
+  one retry on a fresh pool; persistent failures still flow into the
+  standard `failed/` move so a single broken file can't pin the
+  watcher in retry loops.
 
 ## [0.1.1] - 2026-05-13
 

hyperdb-mcp/Cargo.toml

@@ -41,6 +41,7 @@ notify = "8"
 tokio-util = { version = "0.7", features = ["rt"] }
 tempfile = { workspace = true }
 sqlformat = "0.5.0"
+dirs = "5"
 
 [target.'cfg(unix)'.dependencies]
 libc = "0.2"

hyperdb-mcp/DEVELOPMENT.md

@@ -196,12 +196,16 @@ Derived fields are merged into the serialized JSON so callers get a self-contain
 
 ---
 
-## Workspace Modes Internals
+## Two-Database Engine Model
 
-- **Ephemeral** — `Engine::new(None)` creates a temp directory (`$TMPDIR/hyperdb-mcp-<pid>/`) with a `workspace.hyper` file. Cleaned up on process exit. In daemon mode, `Engine::drop` issues `DETACH DATABASE` (releasing Hyper's file lock — required on Windows) before `remove_dir_all` deletes the temp directory.
-- **Persistent** — `Engine::new(Some(path))` uses the caller-supplied path. Parent directories are created automatically. `~` is expanded via `$HOME` (no shell crate dependency).
+Every `Engine` holds:
 
-Logs (both `hyperd` server logs and the MCP client log) land in the same directory as the workspace file. The `status` tool reports log paths so operators know where to look.
+- **Ephemeral primary** at `$TMPDIR/hyperdb-mcp-<pid>-<seq>/scratch.hyper`. Created fresh on each `Engine::new` (`<seq>` is a process-wide atomic so parallel test runners and restart-after-ConnectionLost reconnects get distinct files). The connection is *bound* to this database — unqualified SQL routes here. `Engine::drop` always deletes the per-engine temp directory.
+- **Persistent attachment** under the alias `"persistent"`. When `Engine::new(Some(path))` is called, the engine runs `CREATE DATABASE IF NOT EXISTS '<path>'` (creating the file if missing) and `ATTACH DATABASE '<path>' AS "persistent"`. Then `SET schema_search_path = '<primary_db>'` keeps unqualified resolution routing into the ephemeral primary. `Engine::new(None)` skips this step (the `--ephemeral-only` mode); `engine.has_persistent()` reflects which mode was selected.
+
+Catalog and saved-queries meta-tables live in the persistent attachment, qualified via `"persistent"."public"."_table_catalog"` etc. When the engine has no persistent attachment, those operations no-op. See `crate::table_catalog::qualified_catalog` and `crate::saved_queries::WorkspaceStore::qualified_table` for the routing helpers.
+
+Logs land next to the persistent file when one is supplied (so users find them in a stable per-project location); ephemeral-only sessions log to `$TMPDIR/hyperdb-mcp-<pid>/`. The `status` tool reports `ephemeral_path`, `persistent_path`, and `has_persistent` so operators can confirm where data lives.
 
 ## Daemon Mode Internals
 
@@ -239,7 +243,7 @@ Two new code paths fire `report_hyperd_error_to_daemon` (best-effort, 200ms time
 ### Known limitations
 
 - **Hung-but-alive `hyperd`** (TCP listening, but unresponsive to queries) is NOT detected. The monitor's `try_wait()` returns `None` for a hung process; client tool calls hang on the read side without producing a `ConnectionLost` error. Operator recovery is `hyperdb-mcp daemon stop` followed by reconnect.
-- **Watchers (background tasks holding `AsyncConnection` handles in `WatcherRegistry`)** do not currently auto-reconnect after a hyperd restart. They will go quiet until the user re-issues `watch_directory`.
+- **Watchers** auto-recover from hyperd restarts: when an ingest fails with a connection-lost error, the watcher rebuilds its connection pool against the engine's current endpoint and retries the file once. Persistent failures (the second attempt also fails) fall through to the standard `failed/` move so a single broken file can't keep the watcher pinned in retry loops.
 
 See `src/daemon/{mod,discovery,health,run,spawn}.rs` for the full implementation.