perf(worker_queue): partial index + two-phase fetch_work_item dequeue

[iemejia]

Why

Workers dequeue the oldest task no one else is working on. To do that today, PostgreSQL walks past every task already claimed by another worker, one by one. The more workers you run, the more claimed tasks pile up at the head of the queue, so each dequeue gets slower as you scale out.

Cause: the query uses (lock_token IS NULL OR locked_until <= now) with FOR UPDATE, which disallows bitmap index combining, so PostgreSQL scans the primary key in id order and filters claimed rows by hand — O(rows claimed at the head) per dequeue. At a 200k claimed-row stress point that is ~14.6 ms/dequeue.

Fix

Migration 0022 adds a partial index over only the rows still available to claim:

CREATE INDEX idx_worker_ready ON worker_queue (id) WHERE lock_token IS NULL;

fetch_work_item now runs a hot path over that index, falling back to the original query only when no fresh work exists (to reclaim rows whose lease has expired).

End-to-end A/B over 2000 real fetch_work_item calls (200k claimed + 10k available): 14.56 ms -> 0.06 ms per dequeue (~244x); index 176 KB vs 14 MB.

Notes

• Behavioral change: when leases have expired, fresh work is handed out before expired-lease reclaim rather than in strict id order. Leases only expire after a worker crash/stall, and SKIP LOCKED already makes cross-worker ordering best-effort, so this stays within existing guarantees.
• Scope: worker_queue only; orchestrator_queue uses a different eligibility model and is left as separate future work. A reclaim-sweep alternative that preserves exact ordering is described in the 0022 migration header.
• Verified: full suite passes (basic; provider incl. session, tag-filtering, lock-expiration, queue-semantics; regression; session e2e; stress). Migration applies via both psql and the crate migration runner.
• Reproduce the A/B with scripts/bench-dequeue.sh (builds a 0021 vs 0022 schema from the migration files and times real fetch_work_item calls).

[iemejia]

@pinodeca PTAL this is mostly an scalability improvement. The index does not seem to be so big, so it may make sense. PTAL