wesm · webgress · Apr 13, 2026 · Apr 13, 2026 · Apr 13, 2026 · Apr 13, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -133,8 +133,18 @@ Core tables:
 - `sync_runs` / `sync_checkpoints` - Sync state for resumability
 
 Schema files in `internal/store/`:
-- `schema.sql` - Core SQLite schema
-- `schema_sqlite.sql` - FTS5 virtual table
+- `schema.sql` - Core schema (SQLite; shared structure)
+- `schema_sqlite.sql` - SQLite FTS5 virtual table
+- `schema_pg.sql` - PostgreSQL tsvector column + GIN index (opt-in, scaffold)
+
+**Database backend**: SQLite is the default. PostgreSQL support is
+scaffolded behind a `Dialect` interface (`internal/store/dialect.go`);
+see `docs/PG_STATUS.md` for the current state and follow-up
+work required to make PostgreSQL functional end-to-end.
+
+**Test env**: set `MSGVAULT_TEST_DB=postgres://...` to run the store test
+suite against PostgreSQL instead of SQLite (`make test-pg`). Each test
+uses an isolated schema.
 
 ## Parquet Analytics
 
@@ -201,7 +211,8 @@ automatically:
 
 ```bash
 make install-hooks             # Install pre-commit hook via prek
-make test                      # Run tests
+make test                      # Run tests (SQLite default)
+make test-pg                   # Run tests against PostgreSQL (requires MSGVAULT_TEST_DB)
 make fmt                       # Format code (go fmt)
 make lint                      # Run linter (auto-fix)
 make lint-ci                   # Run linter (CI, no auto-fix)

diff --git a/Makefile b/Makefile
@@ -58,6 +58,15 @@ test:
 test-v:
 	go test -tags "$(BUILD_TAGS)" -v ./...
 
+# Run tests against PostgreSQL (set MSGVAULT_TEST_DB first)
+# Example: MSGVAULT_TEST_DB=postgres://user:pass@localhost:5432/db make test-pg
+test-pg:
+	@if [ -z "$$MSGVAULT_TEST_DB" ]; then \
+		echo "MSGVAULT_TEST_DB must be set, e.g., postgres://user:pass@localhost:5432/db" >&2; \
+		exit 1; \
+	fi
+	go test -tags fts5 ./...
+
 # Format code
 fmt:
 	go fmt ./...

diff --git a/docs/PG_STATUS.md b/docs/PG_STATUS.md
@@ -0,0 +1,119 @@
+# PostgreSQL Backend Status
+
+This document tracks the state of PostgreSQL backend support in msgvault.
+
+## Summary
+
+PR1 (tag `pr1-dialect-extraction`) extracted all SQLite-specific behavior
+behind a `Dialect` interface. Zero functional change; SQLite is still the
+default and only production-ready backend.
+
+PR2 (tag `pr2-postgresql-dialect`) adds **foundational scaffolding** for
+PostgreSQL support:
+
+- `PostgreSQLDialect` implementing the `Dialect` interface
+- `pgx` driver wired into `store.Open()` for `postgres://` URLs
+- `schema_pg.sql` with tsvector FTS column and GIN index
+- `PostgreSQLEngine` scaffold parallel to `SQLiteEngine`
+- Dual-backend test harness via `MSGVAULT_TEST_DB`
+- Unit tests for dialect string methods
+
+**PostgreSQL is NOT functionally usable yet.** The work below must complete
+before a PostgreSQL connection can successfully insert a single row.
+
+## What Works
+
+- `PostgreSQLDialect.Rebind()` correctly converts `?` → `$1, $2, ...`
+  (including quoted-string safety)
+- `PostgreSQLDialect.Now()`, `InsertOrIgnore()` (complete + prefix),
+  `InsertOrIgnoreSuffix()`, `FTSSearchClause()`, `UpdateOrIgnore()`
+- `PostgreSQLDialect` error-code classification (23505, 42701, 42P01)
+- `Open("postgres://...")` establishes a connection with pool settings
+- `OpenReadOnly` for PostgreSQL enforces `default_transaction_read_only=on`
+  via pgx `RuntimeParams` (set on every pooled connection at startup)
+- Unit tests for dialect string methods pass without a live Postgres
+- SQLite regression: all existing tests pass unmodified
+
+## Follow-Up Work (Required for PostgreSQL to Actually Work)
+
+### Blockers (schema will not load, no row can be inserted)
+
+1. **Schema type translation**: `schema.sql` uses SQLite-specific types
+   (`DATETIME`, `BLOB`) and `INTEGER PRIMARY KEY` which is not
+   auto-incrementing in PostgreSQL. Options:
+   - Create a dedicated `schema_pg.sql` with PostgreSQL-native DDL
+     (`TIMESTAMPTZ`, `BYTEA`, `BIGINT GENERATED ALWAYS AS IDENTITY`)
+   - Parameterize the shared schema via dialect type mappings
+   - Translate at load time
+
+2. **Thread `Rebind()` through all queries**: Most `s.db.Exec` / `QueryRow`
+   calls in the store layer still pass raw `?` placeholders. pgx rejects
+   these. Affected files: `messages.go`, `sync.go`, `sources.go`,
+   `sources_oauthapp.go`, `api.go`. (`inspect.go` already uses `Rebind()`.)
+
+3. **`queryInChunks` / `insertInChunks` bypass the dialect**: These
+   helpers in `store.go` hardcode `?` placeholders. They need to accept
+   (or be wrapped by) a rebinder.
+
+4. **`LastInsertId()` is not supported by pgx**: Call sites in
+   `messages.go` (EnsureConversation, EnsureParticipant, etc.) and
+   `sync.go` (StartSync, GetOrCreateSource) must be rewritten to use
+   `RETURNING id` (the pattern already exists in `upsertMessageWith`).
+
+5. **Mixed placeholder styles in search**: `api.go:SearchMessages` now
+   builds queries with `$1` from `FTSSearchClause` but still appends
+   `LIMIT ? OFFSET ?`. Must pick one style consistently.
+
+### Issues (correctness/behavior differences)
+
+6. **`FTSBackfillBatchSQL` INNER vs LEFT JOIN**: PostgreSQL version uses
+   inner join on `message_bodies`; SQLite uses LEFT JOIN. Messages with
+   no body row are not indexed on PostgreSQL.
+
+7. **`SET statement_timeout` runs on only one pool connection**: Move to
+   pgx connection string or `AfterConnect` hook.
+
+8. **(Resolved)** `openPostgresReadOnly` now sets
+   `default_transaction_read_only=on` via pgx `RuntimeParams`, so the
+   parameter is applied during the startup packet of every pooled
+   connection rather than once via `db.Exec("SET …")`. The same pattern
+   should be used for `statement_timeout` (item #7).
+
+9. **`GetStats` calls `os.Stat(s.dbPath)`**: For PostgreSQL, `dbPath` is
+   a URL, not a file. `DatabaseSize` silently reports 0. Either skip
+   or query `pg_database_size(current_database())`.
+
+10. **`PostgreSQLEngine` returns `ErrNotImplemented` for most methods**:
+    TUI/MCP/HTTP API will not work against PG. Aggregate, Search,
+    SearchFast, GetGmailIDsByFilter, ListMessages all need parameterized
+    query builders (currently SQLite-specific: `strftime`, FTS5 MATCH).
+
+11. **FTS weight differences**: PostgreSQL applies `setweight('A')` to
+    subject and `'B'` to sender. SQLite FTS5 has no weighting. Ranking
+    results will differ between backends.
+
+12. **`PostgreSQLEngine` is not constructed anywhere**: TUI/API/MCP
+    still build `SQLiteEngine` unconditionally.
+
+## Running Tests Against PostgreSQL
+
+Once blockers above are resolved:
+
+```bash
+# Start a PostgreSQL instance, then:
+export MSGVAULT_TEST_DB=postgres://user:pass@localhost:5432/msgvault_test
+make test
+```
+
+Each test creates and drops its own schema (`msgvault_test_<hex>`) for
+isolation. The `testutil.NewTestStore()` helper detects the env var and
+routes accordingly. If `MSGVAULT_TEST_DB` is unset, SQLite is used (default).
+
+## Why Ship Scaffolding?
+
+The `Dialect` abstraction + scaffolded PostgreSQL implementation lets the
+remaining work proceed incrementally without further disrupting the
+SQLite path. The interface design has been validated end-to-end
+(SQLiteDialect produces identical SQL; unit tests confirm PostgreSQLDialect
+generates valid PostgreSQL SQL). Future PRs can tackle the follow-up work
+file-by-file.
diff --git a/flake.nix b/flake.nix
@@ -27,7 +27,7 @@
           pname = "msgvault";
           version = "0.14.1";
           src = ./.;
-          vendorHash = "sha256-cY8Ooixv9GQtOsryCtWdK6iCqzMCK1/x/26/TLJ5+bs=";
+          vendorHash = "sha256-KNtjVBp4OXz2K4dyj/8unnhISddRk8ir3PWvhEsbnUs=";
           proxyVendor = true;
           subPackages = [ "cmd/msgvault" ];
           tags = [ "fts5" ];

@@ -16,6 +16,7 @@ require (
 	github.com/go-chi/chi/v5 v5.2.5
 	github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f
 	github.com/google/go-cmp v0.7.0
+	github.com/jackc/pgx/v5 v5.9.1
 	github.com/jhillyerd/enmime v1.3.0
 	github.com/marcboeker/go-duckdb v1.8.5
 	github.com/mark3labs/mcp-go v0.48.0
@@ -58,6 +59,9 @@ require (
 	github.com/google/jsonschema-go v0.4.2 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jackc/pgpassfile v1.0.0 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
+	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056 // indirect
 	github.com/klauspost/compress v1.17.11 // indirect
 	github.com/klauspost/cpuid/v2 v2.2.9 // indirect

@@ -59,6 +59,7 @@ github.com/coreos/go-oidc/v3 v3.18.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
 github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
@@ -97,6 +98,14 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
+github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
+github.com/jackc/pgx/v5 v5.9.1 h1:uwrxJXBnx76nyISkhr33kQLlUqjv7et7b9FjCen/tdc=
+github.com/jackc/pgx/v5 v5.9.1/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
+github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
+github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
 github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056 h1:iCHtR9CQyktQ5+f3dMVZfwD2KWJUgm7M0gdL9NGr8KA=
 github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
@@ -162,8 +171,11 @@ github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
 github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf h1:pvbZ0lM0XWPBqUKqFU8cmavspvIl9nulOYwdy6IFRRo=
 github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf/go.mod h1:RJID2RhlZKId02nZ62WenDCkgHFerpIOmW0iT7GKmXM=
-github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
-github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
@@ -231,6 +243,7 @@ gonum.org/v1/gonum v0.15.1 h1:FNy7N6OUZVUaWG9pTiD+jlhdQ3lMP+/LcTpJ6+a8sQ0=
 gonum.org/v1/gonum v0.15.1/go.mod h1:eZTZuRFrzu5pcyjN5wJhcIhnUdNijYxX1T2IcrOGY0o=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v1 v1.0.0-20140924161607-9f9df34309c0/go.mod h1:WDnlLJ4WF5VGsH/HVa3CI79GS0ol3YnhVnKP89i0kNg=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 howett.net/plist v1.0.1 h1:37GdZ8tP09Q35o9ych3ehygcsL+HqKSwzctveSlarvM=