Skip to content

fix(db): roll the events/delivery_log catch-all partition forward - #2456

Open
taehalim wants to merge 1 commit into
block:mainfrom
taehalim:fix/partition-roll-forward
Open

fix(db): roll the events/delivery_log catch-all partition forward#2456
taehalim wants to merge 1 commit into
block:mainfrom
taehalim:fix/partition-roll-forward

Conversation

@taehalim

Copy link
Copy Markdown

Problem

Fixes #2396.

The initial schema creates monthly partitions only through 2026-06 plus a *_p_future catch-all from 2026-07-01, and ensure_future_partitions (already called at relay startup) treated "month covered by an overlapping partition" as success. So every write since July lands in the catch-all — it grows without bound and range pruning degrades to seq scans. The manual staging surgery only buys time until 2027-01-01, on every deployment.

Fix

Option 1 from the issue — a durable roller inside the relay, no new extension dependency.

ensure_future_partitions now distinguishes covered months from ensured ones. When a target month is still absorbed by the catch-all, it advances the catch-all in one transaction:

  1. DETACH the catch-all (writers queue behind the ACCESS EXCLUSIVE lock from here to commit — DDL plus at most one validation scan).
  2. Empty → drop it. All rows in one calendar month → re-attach it as that month's partition (zero row movement; the attach scan validates bounds — the same shape as the staging surgery, minus the pre-validated CHECK optimization, trading a short scan for simpler code and no constraint-violation window near month boundaries). Rows spanning multiple months → refuse, roll back (layout untouched), and point at the operator procedure in events partitions end at 2026-06 + catch-all; nothing rolls them forward (recurs 2027-01-01) #2396.
  3. Create real monthly partitions for the range the catch-all used to cover.
  4. Recreate the catch-all above the horizon (now + months_ahead + 1).

The roll is idempotent and serialized across relay instances by a transaction-scoped advisory lock (pg_try_advisory_xact_lock); losing the race is success. delivery_log is partitioned on delivered_at, not created_at, so the allowlist now carries the partition-key column per table.

The relay also re-runs the roll on a periodic tick (BUZZ_PARTITION_ROLL_INTERVAL_SECS, default 6h), so long-running processes cross month boundaries without relying on restarts.

Testing

  • Unit tests for the pure pieces: month arithmetic across year rollovers, partition-bound expression parsing (timestamptz/date literals, MINVALUE, garbage), name/suffix/date validation.
  • Four Postgres-backed tests (#[ignore = "requires Postgres"]), each running in a throwaway schema with search_path pinned so nothing touches real tables: empty-catch-all roll creates the monthly partitions and fresh writes route to them (verified via tableoid); rows absorbed by the catch-all are preserved and served from the re-attached monthly partition; multi-month spans are refused and the rollback leaves the catch-all attached with all rows readable; a second roll in the same month is a no-op.
  • Verified end-to-end against a live dev relay whose DB was in the exact reported state (catch-all holding this month's rows): startup rolled events (rows re-attached as events_p2026_07, _08_10 created, catch-all moved to 2026-11-01) and delivery_log (empty path), and a restart after the roll was a clean no-op.

The initial schema creates monthly partitions only through 2026-06 plus a
p_future catch-all from 2026-07-01, and ensure_future_partitions treated
'month already covered by an overlapping partition' as success. Every
write since July has therefore landed in the catch-all: it grows without
bound and range pruning degrades to seq scans (issue block#2396). The manual
staging surgery only bought time until 2027-01-01.

ensure_future_partitions now distinguishes covered months from ensured
ones and rolls the catch-all forward in one transaction: detach, then
drop it when empty or re-attach it as the single month its rows span (no
row movement — the attach scan validates), create real monthly partitions
for the range it used to cover, and recreate the catch-all above the
horizon. Catch-alls holding rows across multiple months are refused with
a pointer to the operator procedure in block#2396 and the rollback leaves the
layout untouched. The roll is idempotent and serialized across relay
instances with a transaction-scoped advisory lock.

The relay now also re-runs the roll on a periodic tick
(BUZZ_PARTITION_ROLL_INTERVAL_SECS, default 6h), so long-running
processes cross month boundaries safely instead of relying on restarts.

delivery_log is partitioned on delivered_at, not created_at — the roll
resolves the partition-key column per table from the allowlist.

Fixes block#2396

Signed-off-by: taeha <uv.taeha@gmail.com>
@taehalim

Copy link
Copy Markdown
Author

@tlongwell-block this implements option 1 from #2396 (in-relay roller + a startup pass and a periodic tick) — since you filed the issue you're likely the right reviewer. Happy to adjust the approach or rebase if the tree has moved. No rush given everything in flight.

@SeanGearin

Copy link
Copy Markdown
Contributor

Independent confirmation of the diagnosis, for whatever it's worth — we hit this from a code audit before finding #2396. 0001_initial_schema.sql creates events_p_future and delivery_log_p_future as FROM ('2026-07-01') TO (MAXVALUE), and the pre-PR ensure_partition path swallowed the 42P17 overlap as success, so every monthly CREATE from July forward silently no-opped. Matches the production evidence in the issue exactly.

The design holds up against the failure modes I'd worry about. Single-transaction detach → (drop | re-attach | refuse) keeps the layout intact on any bail, since the rollback re-attaches the catch-all untouched. pg_try_advisory_xact_lock scoped to current_schema() with lose-the-race-as-success handles multi-replica cleanly without isolated test schemas contending. Carrying the partition-key column per table in the allowlist is what catches the delivery_log/delivered_at asymmetry a created_at assumption would have missed. And roll_forward_bails_on_multi_month_catchall_rows asserting the error plus the post-rollback row count is the test that actually matters here.

One timing point worth flagging for reviewers, since it isn't visible from the diff: the zero-surgery path depends on this landing before 2026-08-01. Today every stranded row is from July, so min_at and max_at land in the same calendar month and the catch-all is re-attached as events_p2026_07 with no row movement — the attach scan validates and nothing is copied. Once August writes land in that same catch-all, the span check sees two months and correctly returns the "run the operator partition-surgery procedure from issue #2396" error instead. That moves every affected deployment, including the one in #2396, off the automated roll and onto the manual procedure. The PR is strictly less useful after the month boundary than before it.

Happy to test against another dirty-state DB if that's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

events partitions end at 2026-06 + catch-all; nothing rolls them forward (recurs 2027-01-01)

2 participants