feat(clients/typescript): experimental cooperative consumers#214
Conversation
Wraps the cooperative SQL surface (subscribe_subconsumer, unsubscribe_subconsumer, receive_coop, touch_subconsumer) on Client and extends the high-level Consumer with a `subconsumer` / `deadInterval` option that routes the poll loop through receiveCoop. Marked experimental in 0.2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Manual end-to-end script that registers two subconsumers under one logical consumer, sends N events across multiple tick windows, and asserts the worker totals sum to N with zero msg_id overlap. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a TS README section covering subscribeSubconsumer, unsubscribeSubconsumer, receiveCoop, touchSubconsumer, and the subconsumer / deadInterval consumer options. Updates the parity matrix to mark TypeScript as the first client with cooperative consumer coverage. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- bench/coop_demo.ts: two-worker walkthrough referenced from the README's
experimental coop section. Connects via PGQUE_TEST_DSN, partitions a
fixed event set across `worker-1` and `worker-2`, and prints per-worker
message lines plus a summary.
- bench/coop_manual_evidence.ts: scripted runner for the four manual-
evidence scenarios — disjoint partitioning, idempotent
subscribeSubconsumer, active-batch rejection + batchHandling=1 retry
routing, and stale takeover via deadInterval.
- bench/coop_scaling.ts: cooperative scaling benchmark sweeping
--subconsumers (default 1), with --events / --payload / --runs /
--handler-work-ms flags. Pre-publishes events with one tick per chunk
so the allocator has multiple windows to hand out, then runs N async
workers under a shared pg.Pool sized to subN * 2 + 4. Reports CSV
(subconsumers, events_per_sec, seconds) on stdout.
- bench/coop_scaling.sh: driver that runs the full {1,2,4,8,16} sweep,
pipes the CSV to plot.py, and writes coop_scaling.png. Defaults
handler_work_ms to 0.25 — enough work per message to let parallel
workers overlap (so the curve rises 1->4) while keeping the
cooperative FOR UPDATE allocator on the hot path so the plateau is
visible inside {1..16}.
- bench/plot.py: matplotlib renderer (~800x500 PNG) with a footer that
reports the PG version, machine, event count, payload size, and
handler work per message via env vars.
- bench/coop_scaling.png: committed headline chart from a real run on
PG 18.3 / Darwin arm64; 5000 events / 64 byte payload / 0.25 ms
handler work / 3 runs / median.
- .gitignore: drop the per-run CSV; the PNG is the canonical result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a short scaling section to the experimental coop block that references bench/coop_demo.ts (runnable two-worker walkthrough), embeds bench/coop_scaling.png as the headline chart, and explains the rise / plateau / regression shape: parallel handlers overlap on the ascending side, the FOR UPDATE on the cooperative main row dominates on the flat side. Includes the load-bearing footnote that adding more *normal* consumers does not share work — each subscribe is a separate fan-out cursor that re-delivers every event, so coop subconsumers are the way to split work across N workers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scaling benchmark was misleading: Node's single event loop made the measurement reflect async-task interleaving with sub-ms busy spin rather than real cooperative-allocator scaling. A unified bench across all three drivers will land separately. Removing the script, runner, chart, plot helper, and the manual-evidence helper that paired with it. The two-worker demo (coop_demo.ts) stays.
Remove the throughput-scaling subsection, chart image, and reproduction pointers from the experimental cooperative consumers docs. The underlying bench was misleading (see preceding commit). Keep the experimental caveat, API table, and the two-worker demo pointer.
REV Code Review Report
POTENTIAL ISSUES (4)Issues with moderate confidence (4-7/10). Review manually — may be false positives. MEDIUM
MEDIUM
INFO
INFO
Summary
REV-assisted review (AI analysis by postgres-ai/rev) |
Summary
subscribeSubconsumer,unsubscribeSubconsumer,receiveCoop,touchSubconsumeronClient, and asubconsumer/deadIntervaloption onnewConsumerthat routes the poll loop throughreceiveCoop.deadIntervalwithoutsubconsumerthrows at construction.clients/README.mdto flag TypeScript as the first client with cooperative coverage.Test plan
bun run check— typecheck cleanPGQUE_TEST_DSN=postgres://nik@localhost/pgque_coop_ts bun run test— 69 tests pass (10 new intest/coop.test.ts):subscribeSubconsumerreturns 1 then 0receiveCoopround-trips messages andackfinishes the batchmsg_iddeliveryunsubscribeSubconsumerdefault raises on active batchunsubscribeSubconsumer({ batchHandling: 1 })routes the active batch through retrytouchSubconsumerreturns 1 on a registered rowConsumerwith{ subconsumer }dispatches handler and acks via the cooperative pathConsumerwithoutsubconsumeris unchanged (still callsreceive)newConsumer({ deadInterval })withoutsubconsumerthrows with a clear messagebun src/coop_e2e.ts):{ "total_sent": 50, "worker_1_processed": 30, "worker_2_processed": 20, "sum": 50, "overlap_count": 0, "disjoint": true }Notes
touchSubconsumer. README points to it as the long-handler escape valve.teardownCoopTestQueuehelper that nukes subscription / retry / dead-letter rows beforedrop_queue, becausedrop_queue(..., true)callsunregister_consumerwhich now refuses cooperative mains with active members.🤖 Generated with Claude Code