Claim cursors: graveyard-resistant polling on PostgreSQL

[jpcamara]

Solves the PostgreSQL dead-tuple pathology: long-running queries pin the xmin horizon, vacuum can't reap solid_queue_ready_executions, and every poll re-scans the graveyard at the low end of the polling index — CPU proportional to how long the horizon stays pinned. Claim cursors let polls seek past the graveyard instead of scanning through it.

Design

One claim cursor per queue key: each process remembers the last (priority, id) position it observed. Fast polls make a single lexicographic row-constructor seek — WHERE (priority, id) > (?, ?) ORDER BY priority, id — which descends the polling index past the entire graveyard in one shot, across all priorities. When a seek comes up empty, the position is cleared and polls skip the claim query entirely until discovery is due.

Discovery is fundamental, not a safety net: the cursor is not a lower bound for live work — a competing claim of a lower id can roll back after this process advanced past it, sequences aren't commit-ordered so a row can allocate a lower id and commit late, and an empty SKIP LOCKED seek can mean the remaining rows were merely locked by a peer. Cursor-free discovery therefore runs on a time-based cadence and comes in two forms:

• Floored pass (claim_cursors_discovery_interval, 1s default): never asks the database to order by priority. It reads a bounded window of region rows (id > the floor: the highest id this process had observed before the last full pass) in id order over the new (queue_name, id) index (typically a range seek from the floor; a degenerate-statistics plan caps at one heap pass, still far below the classic per-tuple scan), orders the window in memory, and locks its picks by bare id equality with no ORDER BY, a shape that gives no plan an ordering reason to walk an index through the graveyard. When the planner seeks, cost tracks rows enqueued since the last full pass and a drained region costs single-digit buffers; a sequential-scan plan caps at one heap pass per short cadence, which scales with table size — validate with EXPLAIN (ANALYZE, BUFFERS) in production. Every newly enqueued row lands above the floor, so pickup latency for fresh work — including strictly-higher-priority arrivals below the cursor — stays bounded by the short cadence at any polling interval, for rows within the window. A sweep that fills its claim limit leaves discovery due, so a higher-priority backlog below the cursor drains ahead of lower-priority work above it. A region larger than the window (500 rows) keeps (priority, id) order only within the window until it slides or the full pass reaches past it — an explicit, bounded, tested slack.
• Full pass (claim_cursors_full_discovery_interval, 10s default, jittered downward so co-booted workers don't scan in lockstep; 0 disables the split): the only pass that reaches rows at or below the floor (rollback overshoot, late commits, anomalous ids), and the only one allowed to move a cursor, healing stragglers within one full interval.

The floor is process memory — the highest id observed before the last full pass — so nothing is read from catalogs, sequences, or settings. A cached or rewound sequence merely delays the affected row until the next full pass, like a late commit; a freshly booted process runs full passes until it has observed the stream.

Honest framing: the full pass still scans the graveyard once per full cadence (10s default), so polling is graveyard-resistant, not dead-tuple-immune — that cost is amortized, not eliminated. The other trade: a strictly-higher-priority arrival waits for the next discovery (≤ ~1s), inside the slack FOR UPDATE SKIP LOCKED already implies.

Ordering change (deliberate): within a priority, claims follow readiness order (id) instead of enqueue order (job_id). Observable difference: a scheduled job coming due queues behind already-waiting same-priority jobs rather than jumping ahead of them. Polling indexes gain (priority, id), (queue_name, priority, id), and (queue_name, id), added alongside the old pair; the README includes the concurrent migration for existing installs.

PostgreSQL only: row-constructor index seeks and the motivating pathology are PostgreSQL-specific. Other adapters keep the classic path untouched. config.solid_queue.claim_cursors = false disables.

Benchmark

Pinned horizon, 200k unremovable dead tuples in ready_executions (confirmed via VACUUM VERBOSE, heap decorrelated from id order), real ReadyExecution.claim path:

	buffer hits	time
classic poll (no cursors)	200,303	53.3 ms
full discovery pass (unbounded)	200,283	49.1 ms
floored window over a drained region	7	0.04 ms
fast poll (cursor seek)	~1	0.9 ms

On the reproducible idle-poll benchmark (200k dead tuples, all discovery passes included), idle polls drop from 18.1 ms to 0.53 ms — 34x. Because the frequent pass is cheap and sees all new work, this holds at any polling_interval shorter than the full-discovery cadence — including fleets that poll at 1s, where the short cadence coincides with every idle poll. (Polling at or beyond the full interval makes most passes unbounded.) Validate on your own data with EXPLAIN (ANALYZE, BUFFERS).

Testing

• ClaimCursorsTest (PG-gated) asserts query counts and shapes via a sql.active_record capture: single-seek claiming across priorities, instant same-priority pickup, empty seek clearing the position and suppressing queries, higher-priority arrivals found once discovery is due (deterministic expire_discovery!, no sleeps), overshoot healing via discovery, readiness-order semantics, flag-off classic behavior
• Full suite green on PostgreSQL with cursors live everywhere; SQLite and MySQL green on the classic path
• Tests reset cursor state in the shared setup — process-local state must not outlive the records it was derived from

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.