SIGTERM during sync corrupts pglite (negative postmaster.pid, ensureSchema aborts)

[moekatib]

Summary

Killing a long-running sync run (SIGTERM) while PGlite is mid-transaction leaves the embedded Postgres in an unrecoverable state: postmaster.pid contains a negative PID, and on next invocation mem doctor reports:

backend opens and applies schema  ✗  Aborted()

No automatic recovery — user has to delete the pglite directory or hand-edit postmaster.pid to get back to a working state. Any unflushed data is lost regardless.

Repro

1. one sync run fathom --full-refresh (a profile with enrich so the run is slow)
2. kill -TERM <pid> mid-run (Ctrl-C is trapped by runner.ts signalCleanup → graceful; a real SIGTERM from a process manager is not).
3. Inspect ~/.one/mem/postmaster.pid — negative PID.
4. one --agent mem doctor → Aborted() on ensureSchema step.

Root cause (hypothesis)

PGlite's WASM postmaster doesn't fully flush WAL/shm before the signal handler terminates the worker. The negative PID suggests the file was written with an incomplete value during teardown. @electric-sql/pglite doesn't expose a graceful-shutdown hook we currently call in signalCleanup.

Expected

At minimum: detect corrupt-state (negative PID, failed schema apply) on startup and offer a safe recovery path — either reset the pglite dir (with confirmation) or point at a separate backup. Ideally: call a pglite graceful close before process.exit in signalCleanup.

Notes

• Not blocking. Most users don't SIGTERM a sync.
• Separate from PR feat: unified memory (v1.42.0) #125 (which introduced SIGINT/SIGTERM handling for sync state/lock cleanup — the PGlite layer needs its own graceful close).
• Workaround: rm -rf ~/.one/mem/postmaster.pid (lose the db) or start fresh with rm -rf ~/.one/mem (lose all memories).

[moekatib]

RCA update after investigation — original hypothesis was wrong.

The -42 PID in postmaster.pid is normal for PGlite, not a corruption marker. PGlite is WASM-backed Postgres; the "postmaster" doesn't have a real OS process, so it writes a placeholder negative PID on every healthy open. A first attempt at this fix added stale-pid cleanup (delete the file if PID is non-positive or not running) and would have broken every healthy PGlite startup. Reverted.

The real signal is the Aborted() that surfaces on ensureSchema — it's caused by unflushed WAL, not a lockfile. PGlite's WAL needs a checkpoint on shutdown; if the process is force-killed before the checkpoint, the next open can't replay the WAL cleanly.

Fix shipped in de8a302:

1. Graceful close in sync's signalCleanup (prevention). SIGINT / SIGTERM handler now awaits backend.close() (with a 2s timeout so a stuck close can't block exit). Gives PGlite a chance to checkpoint.

2. Clearer mem doctor diagnostic (recovery). When the schema-apply check fails with Aborted(), doctor now appends the recovery path: rm -rf ~/.one/mem.pglite (loses all memory records).

What's still out of reach:

• SIGKILL and kernel panic — uncatchable by definition. No user-space fix possible.
• Partial WAL corruption — once the dir is in that state, PGlite has no replay tool. Wiping the data dir is the only path.

Closing — both prevention + recovery-UX shipped. If someone hits SIGKILL-induced corruption, mem doctor will point them at rm -rf ~/.one/mem.pglite.