PgQueuer v1.4.0
concurrency_limit now holds when several workers dequeue at the same time (#761).
That is the reason for this release. Most of the other commits were the SQL and test work needed to make the limit a table invariant instead of a count.
Upgrade
Run pgq upgrade before starting 1.4.0 workers:
ALTER TABLE pgqueuer ADD COLUMN IF NOT EXISTS slot BIGINT;
CREATE UNIQUE INDEX IF NOT EXISTS pgqueuer_picked_slot_idx
ON pgqueuer (entrypoint, slot)
WHERE (status = 'picked' AND slot IS NOT NULL);Workers check for both at startup and refuse to boot if they are missing. The index takes a SHARE lock on the queue table; on a large table, run it in a quiet window.
Jobs already picked at migrate time keep slot IS NULL and drain as usual.
Fixes
- Concurrent workers could exceed
concurrency_limit. Two claims could both see “capacity left”, thenSKIP LOCKEDwould take different jobs. Limits are now named seats (0..N-1) enforced by the unique index above. --restart-on-failureactually restarts. A failed worker no longer latches the process-level shutdown event and kills the supervisor loop.
Notes
- All workers must declare the same
concurrency_limitfor an entrypoint. If they disagree, the highest value wins. Job.slotis the seat a limited job holds while picked;Nonefor unlimited work.- Typed call sites of
dequeue()needQueueEntrypoint(...)/QueueManagerId(...)for mypy. Runtime is unchanged.@entrypointandenqueue()still take strings.
Thanks to @khalo-sa for the report.
Full Changelog: v1.3.2...v1.4.0