fix(db): roll the events/delivery_log catch-all partition forward - #2456
fix(db): roll the events/delivery_log catch-all partition forward#2456taehalim wants to merge 1 commit into
Conversation
The initial schema creates monthly partitions only through 2026-06 plus a p_future catch-all from 2026-07-01, and ensure_future_partitions treated 'month already covered by an overlapping partition' as success. Every write since July has therefore landed in the catch-all: it grows without bound and range pruning degrades to seq scans (issue block#2396). The manual staging surgery only bought time until 2027-01-01. ensure_future_partitions now distinguishes covered months from ensured ones and rolls the catch-all forward in one transaction: detach, then drop it when empty or re-attach it as the single month its rows span (no row movement — the attach scan validates), create real monthly partitions for the range it used to cover, and recreate the catch-all above the horizon. Catch-alls holding rows across multiple months are refused with a pointer to the operator procedure in block#2396 and the rollback leaves the layout untouched. The roll is idempotent and serialized across relay instances with a transaction-scoped advisory lock. The relay now also re-runs the roll on a periodic tick (BUZZ_PARTITION_ROLL_INTERVAL_SECS, default 6h), so long-running processes cross month boundaries safely instead of relying on restarts. delivery_log is partitioned on delivered_at, not created_at — the roll resolves the partition-key column per table from the allowlist. Fixes block#2396 Signed-off-by: taeha <uv.taeha@gmail.com>
|
@tlongwell-block this implements option 1 from #2396 (in-relay roller + a startup pass and a periodic tick) — since you filed the issue you're likely the right reviewer. Happy to adjust the approach or rebase if the tree has moved. No rush given everything in flight. |
|
Independent confirmation of the diagnosis, for whatever it's worth — we hit this from a code audit before finding #2396. The design holds up against the failure modes I'd worry about. Single-transaction detach → (drop | re-attach | refuse) keeps the layout intact on any bail, since the rollback re-attaches the catch-all untouched. One timing point worth flagging for reviewers, since it isn't visible from the diff: the zero-surgery path depends on this landing before 2026-08-01. Today every stranded row is from July, so Happy to test against another dirty-state DB if that's useful. |
Problem
Fixes #2396.
The initial schema creates monthly partitions only through 2026-06 plus a
*_p_futurecatch-all from 2026-07-01, andensure_future_partitions(already called at relay startup) treated "month covered by an overlapping partition" as success. So every write since July lands in the catch-all — it grows without bound and range pruning degrades to seq scans. The manual staging surgery only buys time until 2027-01-01, on every deployment.Fix
Option 1 from the issue — a durable roller inside the relay, no new extension dependency.
ensure_future_partitionsnow distinguishes covered months from ensured ones. When a target month is still absorbed by the catch-all, it advances the catch-all in one transaction:DETACHthe catch-all (writers queue behind the ACCESS EXCLUSIVE lock from here to commit — DDL plus at most one validation scan).now + months_ahead + 1).The roll is idempotent and serialized across relay instances by a transaction-scoped advisory lock (
pg_try_advisory_xact_lock); losing the race is success.delivery_logis partitioned ondelivered_at, notcreated_at, so the allowlist now carries the partition-key column per table.The relay also re-runs the roll on a periodic tick (
BUZZ_PARTITION_ROLL_INTERVAL_SECS, default 6h), so long-running processes cross month boundaries without relying on restarts.Testing
MINVALUE, garbage), name/suffix/date validation.#[ignore = "requires Postgres"]), each running in a throwaway schema withsearch_pathpinned so nothing touches real tables: empty-catch-all roll creates the monthly partitions and fresh writes route to them (verified viatableoid); rows absorbed by the catch-all are preserved and served from the re-attached monthly partition; multi-month spans are refused and the rollback leaves the catch-all attached with all rows readable; a second roll in the same month is a no-op.events(rows re-attached asevents_p2026_07,_08–_10created, catch-all moved to 2026-11-01) anddelivery_log(empty path), and a restart after the roll was a clean no-op.