Partition keys (1A) + producer idempotency (1B): spec, brief, draft implementation#295
Conversation
Partition-keys benchmark — v0.8 lease model at the target scale (SPEC R2/R7, S4)Question this answers: does the N-slot + batch-lease design sustain a high-volume multi-tenant profile — >400M events/day, ordered per tenant — and what do read amplification (R2) and rotation pinning (R7) cost in practice? Setup
Headline
Steady state: the design holds the target with a wide margin. ~9.0M events produced and consumed per 30-minute phase; per-slot pending p50 0, p99 ~1,320 events (≈ 0.26 s of production); zero buffer reads (fully cache-resident window); CPU peak 26–31% on 16 cores. Doubling N from 16→32 at the same rate changed steady-state lag not at all. Read amplification (R2)
Every slot scans the full stream and filters server-side, so buffers touched per produced event grow with N — observed sublinear (1.60× for 2× slots), all from shared_buffers at this scale. Practical read: at 5k ev/s, going from 16 to 32 slots costs ~5 CPU points. R2 is real, measurable, and affordable at this profile; it argues for keeping N at the parallelism you need, as the SPEC says. Rotation pinning (R7) — demonstrated, quantifiedSlot 7's worker SIGSTOPped for 8 minutes mid-phase:
Two operational findings worth pinning in the docs:
Dead tuples & bloat
Caveats
Bottom line for the >400M/day question: steady state sustains it at ~26–31% peak CPU on 16 dedicated cores with sub-second per-slot lag and zero event-table bloat, at both N=16 and N=32. The two things that need operational discipline are exactly the two the SPEC already names: keep every slot polled (R7 + §15) and alert on per-slot lag. Raw data: |
Forward-looking: does partition-keys shard? (roadmap note — out of scope for this PR)Recording this because it came up while reviewing the design, and because it's evidence the abstraction is cut at the right joint. Not proposed for this PR — this is Phase-N thinking. It composes cleanly, and here's whyEvery guarantee this feature makes is per-key: G1 (order within a key), G2 (single processor per key), retry affinity, and the send_idem dedup window. There is deliberately no cross-key guarantee — no global order, no cross-key transaction. That is exactly the contract shape that lets Kafka/Kinesis shard: if the unit of guarantee is the key, splitting keys across nodes loses nothing, because nothing was ever promised between keys. So the sharded model is the obvious one — key routes to a shard, shard is a full independent pgque install: Two-level hashing: key picks the shard (level 1), then the existing The one trap worth writing down nowIf both levels use the same hash, shard count S and slot count N must not share factors. With S=8, N=4: every key in shard The three routers
The strongest argumentIf the app's own data is sharded by the same key (the multi-tenant profile in the benchmark almost certainly is), app rows and their queue land on the same shard — preserving the transactional-outbox property: Relation to the benchmark just postedSharding is orthogonal to the read-amplification result above, and complementary: read-amp is the ~N× cost of the slot filter within one install (measured 4.39 → 7.02 buffers/event for N=16 → 32). Sharding scales write/storage/CPU horizontally across nodes; slots give per-key parallelism within a shard. At the volume benchmarked (5k ev/s ≈ 432M/day) a single 16-core node sits at ~30% CPU, so sharding isn't needed here — it's the lever for the 10× beyond, or for spreading storage. The single-reader dispatch idea (R6) is the other answer to read-amp and is independent of sharding. What it would cost pgqueAlmost nothing in core — that's the point:
|
samorev Code Review Report
BLOCKING ISSUES (1)HIGH
Summary
Note:
Review metadatasamorev-assisted review (AI analysis by Tanya301/samorev) |
e62b4a4 to
d0e0d93
Compare
Deep pre-merge review — PR #295 (partition keys 1A + producer idempotency 1B)Reviewed the rebased head What I verified (all green)
Findings[minor] Not a security hole and [minor] [nit] Design-note comment conflates the server fence with the epoch token — [nit] [nit] Bundled DLQ concurrency hardening — Notes (explicitly not findings)
|
|
Addressed the Red/green TDD: added a guard case in New head: b7244ab |
b7244ab to
b6b6575
Compare
|
Diff-size reduction pass: 12,783 → 10,068 additions (50 → 29 files).
Verification on docker
🤖 Generated with Claude Code |
4cd5b9c to
971de88
Compare
Design guidance for external PRs adding pg-boss-style idempotency keys (refs #293) and per-partition serialization. Maps both features onto PgQue's existing layering: sidecar tables (rotation-safe), send wrappers that reduce to insert_event(), consumer-side gating instead of engine changes, maint-cycle expiry to bound bloat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuaYcu1XXsVEpsnLhF1FFu
Records why pgque's producer idempotency is a TTL window (log model, not job queue) and why free-once-processed belongs on the consumer side. Includes prior-art survey (SQS/NATS/Rabbit vs pg-boss/Oban/River/ Graphile/Hatchet, pgmq gap) and the producer GC fork. Internal blueprint; basis for the reply to #293. Not yet pushed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuaYcu1XXsVEpsnLhF1FFu
SamoSpec-format spec (blueprints/partition-keys/SPEC.md) for consumer-side ordered, parallel consumption by partition key (Kafka-partition model: order within a key, parallelism across keys, no per-event state). Adds a self-contained on-brand HTML brief at web/public/briefs/partition-keys.html (served by Pages at /briefs/partition-keys.html on merge to main). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuaYcu1XXsVEpsnLhF1FFu
Re-ground the consumer mechanism after ops/security + QA/testability review: drop the (impossible) cooperative-consumer overlay for N independent slot subscriptions filtering via get_batch_cursor extra_where; restate the guarantee as testable G1/G2/G3; correct the retry rationale (ev_id preserved, ev_txid changes); derive pause from existing retry_queue (no new table); fix send-signature collision, ev_extra1/trigger collision, unstable hashtext, fixed-N invariant, slot/owner definition. Add decisions.md and refresh the HTML brief. Remove superseded IDEMPOTENCY_AND_PARTITIONS.md contributor guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuaYcu1XXsVEpsnLhF1FFu
Relocate IDEMPOTENCY_DESIGN.md -> blueprints/idempotency/DESIGN.md to match the partition-keys/ slug layout; update the cross-reference in the partition spec and refresh the note's footer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuaYcu1XXsVEpsnLhF1FFu
…ardrail Act on review (Max): partition-keys is the ordered-slot feature only; producer TTL dedup becomes a separate send-layer feature. - New blueprints/idempotency/SPEC.md: producer TTL dedup feature spec with the key-scope rule in hard language (key must encode the desired EFFECT, not just the entity), atomic claim+append, (queue, idem_key) scoping, rotation/maintenance GC, orthogonality to partition keys. - Repro: add --tier hazard guardrail proving the version-suppression bug — entity-only key drops the v2 migration (0 inserted); effect-scoped key delivers both waves. Reframe Case 1 as 'producer idempotency + consumer mutual exclusion on a plain queue', not partition keys. - partition-keys SPEC: promote rotation pinning to a first-class risk (R7); point §12 at the new feature spec; roadmap 1A/1B/2/3; v0.6. - Brief: scope to ordered slots, separate-feature framing, key-scope footgun callout, rotation-pin note, roadmap pills; v0.7. Guardrail measured (PG16): tenant key -> v2 0 inserted; tenant:version -> v2 N inserted. PASS.
…s-style bench.ts compares PgQue (append + rotation) against a pg-boss-style mutable job table (insert->update->delete) in the same DB. Measured (PG16): consume throughput pgque ~208k ev/s vs jobq ~40k ev/s (~5x — the per-message UPDATE+DELETE churn is the tax). Under backlog, pgque holds ZERO dead tuples and needs ZERO vacuum; the mutable table accumulates ~2 dead tuples per processed job and, after draining a ~150k backlog, sits at ~300k dead tuples / ~34 MiB (grew while draining) — the pg-boss bloat, reproduced. Adds demo.jobq baseline table. Honest gaps documented: produce is round-trip-bound, and the full rotation reclaim-to-zero curve is a follow-up (PgQ rotation is a multi-period state machine).
…resize Applied after Fabrizio tested the repro, plus an advisor + 4-agent workflow that converged independently. - Correct the receive-lock claim: core DOES coordinate (for update of s returns the same active batch idempotently, two_session_receive_lock.sh); the real hazard is the process->ack gap, not a coordination vacuum (§12). - State the mechanism/policy seam (D8): core SQL owns corruption-capable transitions; clients/users own guarded policy loops. - Reject a member/heartbeat/lease table (D10) — redundant with session-death advisory-lock release + G2 — replaced by core slot_lock_key (D7) + a read-only partition_slot_status view. - Add the connection-pooler caveat (session locks need session-mode/direct connections — the Supabase ICP). - Upgrade R4 to a sanctioned epoch-gated drain-then-cutover online-resize protocol (D9, §15) with retry-flush + DLQ-preservation guards (Kinesis parent-shard-drain shape); immutable N in Phase 1. - Promote "every slot must be polled" to a hard R7-adjacent requirement. - Brief + decisions.md updated; changelog v0.7. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Materialize every historical draft of the partition-keys brief (v0.1–v0.7) as its own self-contained page under web/public/briefs/partition-keys/, extracted from git history. Inject a version-switcher nav into each page and the live brief, add an index landing page. All self-contained (inline CSS, no external assets) so they render under the Pages CSP. - /briefs/partition-keys.html -> live (latest, v0.7) + switcher - /briefs/partition-keys/ -> version index - /briefs/partition-keys/vX.Y.html -> each archived draft Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final review pass caught the self-contradiction v0.7 introduced and a resize data-loss hole; applied: - F1: the receive-lock correction (§12) makes the session claim LOAD-BEARING for G2, not "pure liveness" — fixed G2 (§2), the two-lock framing (§15), D7, the mechanism/policy prose, and the brief lock cards. Claim releases only at a batch boundary (releasing mid-batch hands the successor the same open batch). - F2: reworked online resize from ev_id-gating to tick-window gating — fixes the abort_resize data-loss hole and the ev_seal type conflation; abort is now loss-free by construction; complete_resize takes the subscription row lock; DLQ re-home + retry-flush specified. Marked the protocol "draft — needs its own review round before Phase 3." - F3/F4: D10/pooling sharpened — session lock leaks onto the pooled backend under transaction pooling (wedge, not miss); only the claim connection needs session mode; tcp_keepalives_* for silent partitions. - F6: brief drift (lock cards, resize note, D8/D9 ID collision -> A1/A2). - F7: epoch column gets a job — receive_partitioned returns it as a user-space fencing token. - Regenerated the v0.7 archive snapshot to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fresh-eyes review after the v0.7 refinement: - §8 grants list now includes slot_lock_key/claim_slot/release_slot and select on partition_slot_status (they were core per D7 but ungranted). - Removed review-finding labels (F7/F2) that leaked into spec prose; replaced with real cross-references (D9, §8, §15). - Brief pooling bullet updated to the corrected semantics: the session lock leaks onto the pooled backend (wedge, not miss), and only the claim-holding connection needs session/direct mode. - v0.7 archive snapshot regenerated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HYTw9GXVXJvectNqZ2fqsq
N independent slot consumers over the append-only log (SPEC v0.7 §6/§8/§15): send(queue, type, payload, partition_key) -> ev_extra1; subscribe_slot/ unsubscribe_slot with persisted enforced N; receive_partitioned via the admin-only get_batch_cursor extra_where hash filter (SECURITY DEFINER co-ownership, validated ints only); ack_partitioned; slot_lock_key/ claim_slot/release_slot shared advisory namespace; partition_slot_status view (owner pid via pg_locks, cursor lag). skip policy only — no pause, no resize. Red/green TDD: tests/test_partition_keys.sql (US-12.1..12.7, dblink second backend) failed against plain pgque.sql, green after the feature; tests/two_session_slot_claim.sh proves claim exclusivity + crash reclaim + owner visibility across two real sessions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HYTw9GXVXJvectNqZ2fqsq
pgque.send_idem(queue, type, payload, idem_key, ttl, partition_key) returns (event_id, deduped): atomic single-statement claim on pgque.idem (insert .. on conflict do update .. where expired), append + claim commit atomically, dedup returns the ORIGINAL event_id. maint_idem() GC self-registers via the existing queue_extra_maint hook — pgque.maint() reaps expired claims with no maint.sql edit. Exact-match (queue, idem_key) scoping; key-must-encode-effect hazard documented. Red/green TDD: tests/test_send_idem.sql (US-13.1..13.4 + atomicity + dblink race: two concurrent send_idem same key -> exactly one insert). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HYTw9GXVXJvectNqZ2fqsq
- SPEC §17 (partition-keys) + idempotency SPEC user-story sections: US-12.1..12.7 (Fabrizio case 2: lifecycle events) and US-13.1..13.5 (case 1: migration dedup), each with acceptance criterion + test pointer; brief gets a matching User Stories panel. - tests/acceptance/us12_partition_keys.sql, us13_producer_idempotency.sql wired into run_acceptance.sql; regression tests wired into run_all.sql. - build/transform.sh includes partition_keys.sql + send_idem.sql in the default API surface; sql/pgque.sql + pgque-tle.sql rebuilt. Verified end-to-end on a fresh PG 18.3: install -> full regression (173 PASS) -> full acceptance (US-1..US-13 ALL PASSED) -> two-session manual test (claim exclusivity, crash reclaim, owner visibility). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HYTw9GXVXJvectNqZ2fqsq
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slot ownership moves from session-scoped advisory locks to a batch-granularity lease (worker id + TTL + epoch) in pgque.partition_slot: plain transactional DML, so it works on one PgBouncer transaction-mode pool with no session state and no connection-per-worker floor (Fabrizio review). receive/ack now require and renew the lease (server-enforced G2); takeover of an expired lease bumps the epoch and fences the zombie. slot_lock_key is removed. Spec v0.8 (new D11); adds T-fencing and the previously missing T-retry-affinity test. Verified: tests/run_all.sql, tests/acceptance/run_acceptance.sql, tests/two_session_slot_claim.sh all green on a fresh install (PG local); install idempotent over the old draft via guarded drops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review pass on v0.8 (adversarial SQL + consistency + brief), applied in place without a version bump: - plain receive/ack/nack reject partition slot consumers (#-names): without the guard a reader could drive the raw slot subscription lease-free and unfiltered, and a fenced zombie could double-ack via plain ack(batch_id) - add nack_partitioned: lease-fenced slot retry/DLQ path (plain nack no longer serves slot batches); shared _nack_batch_event core preserves the #98 canonical re-query and #104 idempotent DLQ - spec fixes: epoch is returned by claim_slot (not receive), the D9 resize epoch and D11 lease epoch are distinct counters, partition_block is Phase-2-only, null-key routes to slot 0 (G1), grace-rule wording, G2 enforcement scoped honestly to the pgque-api surface - brief updated to v0.8 (+archived copy, version switcher) - new tests: raw-slot guards, claim validation, view epoch after takeover, lease renewal heartbeat Verified: run_all, acceptance, two_session_slot_claim.sh green on a fresh install of the rebuilt sql/pgque.sql. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The brief named session-scoped pg_try_advisory_lock in the Tier A per-key recipe while the idempotency spec and US-13.5 test use pg_try_advisory_xact_lock -- the xact-scoped variant is pooler-safe and consistent with D11's rejection of session locks. Align the brief (live + v0.8 archive) and spec section 12 wording. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keyed zipfian producer (pgbench), N slot workers on the real v0.8 lease API (bun), per-slot lag / pgss / bloat / sys samplers, three phases (steady-16, steady-32, stalled-slot for R7), summarizer with read-amp and dead-tuple sections. Results from R1 on Hetzner CCX43: benchmark/partition-keys/results/r1-ccx43-summary.md and PR comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the named-prospect ICP/profile framing with neutral 'high-volume multi-tenant' / 'transaction-pooler ICP' wording across the partition-keys spec, decisions, brief (v0.8 + archive), and benchmark harness/summary. Generic managed-provider compatibility lists are unchanged. Anti-leak: no specific prospect in public artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove benchmark/partition-keys/node_modules/ from git (285 files, ~85k lines of the PR diff) and gitignore the path. bun.lock stays committed for reproducible installs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The plain receive/ack/nack guards treat any consumer name containing '#'
as a partition slot consumer ("<consumer>#<slot>/<n>") and refuse it, but
pgque.subscribe did not reject '#' at registration. A plain consumer
registered with a '#' in its name was therefore permanently locked out of
receive/ack/nack. Reject the reserved character in the pgque.subscribe
API wrapper instead of at first use; the core register_consumer engine is
left untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
benchmark/partition-keys/ and blueprints/partition-keys/repro/ now live in PR #297 (claude/pgque-partition-keys-bench), byte-identical. Keeps this PR's diff focused on the feature: SQL, tests, specs, briefs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
receive/ack/nack_partitioned each ran _partition_n, a queue_id lookup, and _touch_lease; merge the three steps into one _slot_guard (same error precedence: wrong-N before lease errors, identical messages) and factor the ack/nack open-batch lookup into _slot_batch. Drops the now-removed helpers on upgrade from a draft install. Behavior unchanged; also saves one queue lookup per consume call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restate the C-style multi-line comment rule and add a comment-discipline rule: comments state a constraint or invariant in a line or two; no user-story/requirement IDs or spec-process artifacts in code, no restating the spec at length. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Convert multi-line -- comment runs to /* */ blocks, drop user-story and spec-artifact tags (US-x.y, I2, G2), and slim the send_idem/self-wire and guard essays to their load-bearing invariants. Comments-only; regenerated sql/pgque.sql and sql/pgque-tle.sql are byte-identical in executable SQL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clarify that per-function/inline comments state one invariant in 1-3 lines while one consolidated C-style design-notes block per file is encouraged, and that these style rules never apply to the inherited PgQ engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Convert multi-line -- runs to /* */ and trim the v0.1-wrapper upgrade essays, per-function headers, and uninstall/DLQ narration to their load-bearing invariants; drop version/issue tags. Comments-only; the regenerated installs change only comment lines. PgQ engine untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebase onto the stable/devel layout: partition-keys + send_idem API, comment-style reformat, and regenerated devel/sql/pgque.sql + pgque-tle.sql. sql/ stays frozen at v0.2.0; all branch source edits land under devel/sql/. Test doc refs repointed at devel/sql/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
971de88 to
500e40c
Compare
samorev Code Review Report
No blocking issues for the 0.3.0-alpha1 cut. Two non-blocking notes and two lower-confidence items below. Review memory
What was verified
NON-BLOCKING (2)LOW
INFO
POTENTIAL ISSUES (2)LOW
INFO
Summary
Note:
Known open items already tracked on this PR were not re-reported (non-superuser-installer T-security variant, D9 online resize, Phase-2 Review metadatasamorev-assisted review (AI analysis by Tanya301/samorev) |
What's here
Design layer (converged through 3 internal review rounds + Max + Fabrizio + two stronger-model refinement passes):
blueprints/partition-keys/SPEC.mdv0.7 — ordered per-key consumption via N slot subscriptions; G1/G2 corrected (session claim is load-bearing, not "pure liveness"); mechanism/policy seam; member/heartbeat table rejected (D10); tick-gated online-resize protocol as sanctioned future path (D9, draft); connection-pooler caveat (only the claim connection needs session mode).blueprints/idempotency/SPEC.md— producer TTL dedup as a separate send-layer feature.User stories — US-12.1–12.7 (lifecycle events: keyed send, per-key order, cross-key parallelism, single processor, claim/crash-recovery, observability, enforced N) and US-13.1–13.5 (migrations: TTL dedup, effect-scoped keys, expiry, GC, mutual-exclusion recipe) in both SPECs, the brief, and executable acceptance tests.
Draft implementation (red/green TDD, engine untouched):
sql/pgque-api/partition_keys.sql— 4-argsend,subscribe_slot/unsubscribe_slot(persisted enforced N),receive_partitioned(SECURITY DEFINER co-ownership over the admin-onlyget_batch_cursorextra_where hook),ack_partitioned,slot_lock_key/claim_slot/release_slot,partition_slot_statusview.sql/pgque-api/send_idem.sql— atomic single-statement TTL claim, dedup returns the original event_id,maint_idem()GC self-registered viaqueue_extra_maint(zero maint.sql edits).build/transform.sh;sql/pgque.sql+pgque-tle.sqlrebuilt.Reproduction & benchmarks (earlier commits): TS/bun repro of both workloads incl. the version-suppression hazard guardrail; bloat-under-backlog + throughput bench vs a pg-boss-style mutable job table.
Verification (fresh PostgreSQL 18.3)
Repro commands:
Known follow-ups before leaving draft
pausepolicy (Phase 2) blocked on O1 — deliberately NOT implemented🤖 Generated with Claude Code
https://claude.ai/code/session_01HYTw9GXVXJvectNqZ2fqsq