Skip to content

v1.4.0

Latest

Choose a tag to compare

@janbjorge janbjorge released this 14 Sep 14:28
· 26 commits to main since this release
dbf9964

PgQueuer v1.4.0

concurrency_limit now holds when several workers dequeue at the same time (#761).

That is the reason for this release. Most of the other commits were the SQL and test work needed to make the limit a table invariant instead of a count.

Upgrade

Run pgq upgrade before starting 1.4.0 workers:

ALTER TABLE pgqueuer ADD COLUMN IF NOT EXISTS slot BIGINT;
CREATE UNIQUE INDEX IF NOT EXISTS pgqueuer_picked_slot_idx
    ON pgqueuer (entrypoint, slot)
    WHERE (status = 'picked' AND slot IS NOT NULL);

Workers check for both at startup and refuse to boot if they are missing. The index takes a SHARE lock on the queue table; on a large table, run it in a quiet window.

Jobs already picked at migrate time keep slot IS NULL and drain as usual.

Fixes

  • Concurrent workers could exceed concurrency_limit. Two claims could both see “capacity left”, then SKIP LOCKED would take different jobs. Limits are now named seats (0..N-1) enforced by the unique index above.
  • --restart-on-failure actually restarts. A failed worker no longer latches the process-level shutdown event and kills the supervisor loop.

Notes

  • All workers must declare the same concurrency_limit for an entrypoint. If they disagree, the highest value wins.
  • Job.slot is the seat a limited job holds while picked; None for unlimited work.
  • Typed call sites of dequeue() need QueueEntrypoint(...) / QueueManagerId(...) for mypy. Runtime is unchanged. @entrypoint and enqueue() still take strings.

Thanks to @khalo-sa for the report.

Full Changelog: v1.3.2...v1.4.0