Bound NIP-RS retention and search indexing

[tlongwell-block]

Summary

• bound storage for conforming NIP-RS read-state events (kind 30078, exactly one d=read-state:<32 lowercase hex> tag, and exactly one exact two-element t=read-state tag) by hard-deleting superseded payloads instead of retaining soft-deleted history
• preserve NIP-33 ordering and stale-replay protection in a compact per-coordinate watermark, including after a later NIP-09 deletion removes the live head
• migrate existing data safely by seeding watermarks first and physically purging only superseded NIP-RS rows that have a strictly dominating live head
• reduce future full-text-search amplification with a positive search_tsv allowlist for kinds 0, 9, 40002, 45001, and 45003
• give fresh empty databases that allowlist automatically, while leaving populated databases on an explicit out-of-band rewrite path

Runtime behavior

For conforming NIP-RS coordinates—requiring exactly one d tag and exactly one exact two-element t=read-state tag—replacement remains serialized by the existing per-coordinate advisory lock. The database trigger advances/checks the durable watermark on every insert—including inserts from pre-PR relay binaries during a rolling deploy—while the application checks both the live event and watermark. An older tuple is rejected (lowest event ID wins equal timestamps), and superseded payloads plus defensive mention rows are physically removed. Legacy soft-delete and NIP-09 paths are also converted to physical deletion by a database trigger, while retaining only the compact ordering watermark.

Other kind-30078 application data and all other NIP-33 kinds keep the existing soft-delete replacement behavior. channel_id remains outside the NIP-33 replacement key.

Deployment behavior

With BUZZ_AUTO_MIGRATE enabled, migration 0007 takes SHARE ROW EXCLUSIVE on events, so reads remain available while event inserts, updates, and deletes wait for commit. It:

1. creates parameterized_event_watermarks and the mention-cleanup index;
2. fails closed if legacy data contains a deleted conforming NIP-RS tuple that outranks its live head;
3. seeds each coordinate's greatest accepted tuple across live and historical rows; and
4. physically deletes only superseded NIP-RS history with a strictly dominating live head.

Migrations 0007 and 0009 are checksum-frozen at their published contents. Additive migration 0009 installs the database triggers that enforce watermark advancement/stale rejection and physical NIP-RS deletion across old and new relay binaries, plus the live-event fence for post-commit mention indexing. Exact duplicate idempotence is enforced at the durable coordinate level: an incoming tuple exactly equal to the watermark is skipped by the insert trigger and reports zero affected rows whether its payload is live or already deleted, eliminating replay/deletion resurrection races without another lock edge. Event replacement and legacy soft-delete cleanup both use the same event-row-before-mentions lock order. Additive migration 0010 replaces only the watermark guard function to make exact equality a zero-row no-op. Additive migration 0011 then tightens every database-side classifier to exact tag cardinality and installs a fail-closed hard-delete fence for mixed-version writers; corrected writers opt in transaction-locally, while old writers attempting an unsafe delete raise and roll back the whole replacement. Databases that already recorded migrations 0001–0010 upgrade directly to 0011 without a sqlx checksum mismatch, and migrations 0007–0010 remain byte-for-byte frozen.

Rows deleted explicitly through NIP-09 and lacking a live head are not purged by 0007. The migration does not rewrite search_tsv or rebuild its GIN index.

Migration 0008 serializes an emptiness check with event writers. A fresh empty database switches to the positive FTS allowlist. A populated database records 0008 as applied without changing its existing denylist expression or rewriting heap/TOAST/GIN storage. Search behavior therefore intentionally differs between fresh and existing installations until maintenance is performed.

Optional maintenance to convert populated databases

The PR is boot-safe without this step: populated databases keep their existing search expression. To convert one to the positive allowlist, run scripts/maintenance/nip_rs_search_allowlist.sql separately in a sized maintenance window. It takes ACCESS EXCLUSIVE, so event reads and writes block; it rewrites the generated search_tsv column and rebuilds the GIN index. Confirm headroom for replacement heap/TOAST/index files plus WAL before running it.

Migration 0007's in-place hard deletes do not by themselves return filesystem space to the volume. The allowlist maintenance script rewrites the generated column and rebuilds the GIN index, replacing those relations, but it needs peak headroom while old and new files coexist and release can be delayed by old snapshots. If cleanup is run without that rewrite—or measured residual bloat remains afterward—use a table repack or VACUUM FULL in a separately sized maintenance window.

Validation

• just ci
• cargo test -p buzz-db
• cargo check -p buzz-db
• Postgres regressions proving replacement hard-deletes the old payload and the watermark rejects replay after simulated NIP-09 deletion
• mixed-version A → old writer B → NIP-09 delete B → replay B and X (A < X < B) regression proving the database advances to B, physically removes old payloads, fences stale post-commit mentions without deadlock, and rejects both exact and intermediate replay
• fresh 0001→0008 migration proving the installed catalog expression is the positive allowlist
• populated-at-0007 migration proving 0008 preserves the legacy expression and leaves all 99 events-family relation file nodes unchanged
• fail-closed rollback test for a deleted tuple outranking its live head
• exact migration/runtime NIP-RS predicate tests covering duplicate and malformed d/t tags, including legacy SQL writes
• fail-closed mixed-version hard-delete regression proving the prior payload and mention survive rollback and transaction-local authorization clears after both commit and rollback
• authentic pre-0007 ambiguity regression proving versions and source data remain unchanged on failure, then migration 0001–0011 succeeds after operator repair
• explicit upgrades from databases with published migrations recorded through 0008, 0009, and 0010 to their additive successors without checksum mismatch
• all 18 Postgres search integration tests against migrations 0001–0009, including a non-vacuous soft-deletion exclusion tripwire
• after merging current origin/main as a merge commit, Hermit-activated pre-push hooks, 79 non-Postgres buzz-db tests, fresh 0001–0011 migration, authentic pre-0007 fail-closed/retry, exact-cardinality, mixed-version replay/lock, and hard-delete fence/GUC regressions are green at 882367357ad29820eb7c4c604991d62b6ebe0bec; GitHub CI pending