v0.6.16

Recommended for everyone on 0.6.15 using queue storage. Rescue drains a large backlog of abandoned claims, which a killed runtime leaves behind and which stops the queue and claim rings from rotating, in minutes instead of the hours or days 0.6.15 took. Batch inserts (insert_many_copy, insert_many_skip_unique_conflicts) give way to per-row producers that share their keys or lanes instead of deadlocking with them, and creating a new queue lane no longer deadlocks between QueueStorage and SQL producers. Operators can durably halt and resume maintenance.prune across the fleet, admin::get_job no longer misses a failed job while prune moves it, and awa migrate re-applies the claim batch index and batch insert patches with fixes.

Upgrading

• The schema stays at v40. Run awa migrate once as the schema owner while traffic keeps running. On a 0.6.15 database it applies three repeatable patches, each in its own transaction and none taking a table lock: claim_batch_job_index again, replacing each queue-storage schema's ensure_claim_batch_job_index; insert_jobs_compat_batch, replacing the body of awa.insert_jobs_compat_batch, so 0.6.15 producers get the deadlock fix as soon as it commits; and the new prune_controls, which adds the awa.runtime_controls and awa.runtime_control_observations tables and the halt and resume functions without altering existing tables. See the upgrade guide.
• The patch does not grant runtime or operator access. Grant runtime roles SELECT on awa.runtime_controls and SELECT, INSERT, UPDATE, DELETE on awa.runtime_control_observations, and an operator role SELECT on both and on awa.runtime_instances plus EXECUTE on awa.halt_runtime_scope and awa.resume_runtime_scope (runtime controls). A 0.6.16 runtime that cannot read the control stops admitting prune within five seconds of the patch committing, while other maintenance continues, so apply the grants before rolling binaries, or expect prune to pause until they are in place.
• External migration runners: apply the new R__claim_batch_job_index.sql and R__insert_jobs_compat_batch.sql, then R__prune_controls.sql, after V40. Replace all R__ files together with the full set from awa migrate --extract-to <dir> --from 40; --pending lists only the patches a database lacks.
• During the roll, a 0.6.15 QueueStorage process that is the first to use a new (queue, priority) lane can still deadlock with another producer of that lane. Create a new queue's lanes ahead of traffic with the 0.6.16 awa storage prepare-queue, or retry the failed transaction.
• 0.6.15 binaries ignore a prune halt and can still become maintenance leader. Upgrade or fence every process that can lead before relying on a halt, and check awa control status.
• Faster rescue needs no migration and starts once the maintenance leader runs 0.6.16. On a fleet with a backlog of abandoned claims, expect a burst of retries as it drains: every rescued attempt counts against max_attempts, so its job goes back to retryable with the usual backoff (2^attempt seconds plus up to 25%, about 2 s for a first attempt) and becomes due at about the rate rescue drains, or fails or moves to the DLQ if that was its last attempt. Closures, retry rows and their WAL arrive at the same rate. Setting rescue_pass_budget or rescue_pass_max_rows to zero keeps the 0.6.15 cadence of one batch per rescue interval.
• Binaries can roll back to 0.6.15 with the patches left in place. A 0.6.15 leader ignores a stored halt. Rolling the maintenance leader back restores the 0.6.15 rescue rate.

Fixed

Rescue and heartbeats

• Queue-storage rescue drains a large backlog of abandoned claims in bounded time (#627). A killed runtime leaves every attempt it claimed open, and until they close, queue prune skips the queue slots they reference (queue.unclosed_claim_refs) and claim prune skips their claim slots, so neither ring can wrap. 0.6.15 rescued at most 500 attempts per kind per 30-second interval, about 33 a second, and its cursor scans reread every claim already swept. A rescue pass now takes 500-attempt batches, each in its own transaction, until a batch comes back short, rescue_pass_budget (default 1 s, at most half the rescue interval) is spent, or it has rescued rescue_pass_max_rows (default 5,000), and while a backlog remains the next pass runs a second later. Cursor scans seek through the index, so a batch costs the same at the end of a drain as at the start, and rescue runs without JIT, whose compilation took seconds per statement. With the defaults, 1.6 million abandoned row-local claims past their deadlines drained in about 17 minutes (1,600 a second) instead of about 13 hours, and both rings moved past the pinned slots about two minutes later. Deadline and callback rescue act on expired deadlines from their first pass; stale-heartbeat rescue takes more than one batch per pass only once runtime snapshots show the claims' owners gone, about 3 to 3.5 minutes after a runtime stops reporting with default settings. See draining a backlog of abandoned claims.
• Heartbeats no longer queue behind a completion. Compact receipt claims are heartbeated by their claim batch and item index, an attempt whose closer holds its lock is skipped rather than waited for, and lease and progress writes skip rows a finalizer has locked and retry them on the next pass. One stalled completion could otherwise hold up a whole heartbeat pass until other attempts went stale and were rescued. A finished attempt stops flushing progress once its handler returns.

Enqueue deadlocks

• insert_many_copy and insert_many_skip_unique_conflicts no longer deadlock with per-row producers (generic insert_many, single-job inserts, 0.6.14 and earlier binaries) that share their unique keys or lanes. Since 0.6.15 the batch function took all of a batch's unique claims before its lanes, so a per-row transaction that held a lane and then inserted a key the batch had claimed waited on the batch while the batch waited on it, and PostgreSQL aborted one of them with deadlock detected (40P01). For its first 2 s the function now limits each lock wait to 500 ms, or half of its own session's deadlock_timeout if lower. A longer wait rolls back the claims and lanes it holds and retries, so the per-row transaction proceeds and the batch then skips, or fails on, the keys that transaction committed. The same covers a transaction that enqueues lanes in separate statements in the opposite order to a batch. This holds when the batch call is the first thing its transaction locks, the per-row session's deadlock_timeout is above 500 ms (PostgreSQL's default is 1 s), and the batch inserts its unique claims within 500 ms: roughly 10,000 to 20,000 keys on a fast server, fewer on a slow or busy one. Results, the caller's lock_timeout on each wait, and the lane lock window are unchanged.
• Creating a new queue lane through QueueStorage (its batch enqueues, and workers promoting scheduled or retried jobs) no longer deadlocks with a concurrent insert, insert_many or insert_many_copy, or a 0.6.14 or 0.6.15 per-row producer, that is also the first to use the same (queue, priority) (#692). QueueStorage took the lane's provisioning lock before writing its queue_lanes row, while SQL inserts write the row first and take the lock in the head-table trigger that creates the lane's sequences, so each waited on the other and one failed with 40P01. QueueStorage now writes the row first too. Lanes that already existed were never affected. No schema change.

Claim batch index

• Claim prune no longer treats a partial GIN index on lease_claim_batches.job_ids, or a session with quote_all_identifiers on, as the claim batch index. It recognises only a valid, non-partial GIN index on job_ids alone. When another index or relation already has the index's name, ensure_claim_batch_job_index leaves it in place, returns false and raises a warning naming it, instead of reporting the slot indexed; the slot keeps scanning until an operator drops or renames it.
• Refreshing ensure_claim_batch_job_index as a migration role that owns it but may only SET ROLE to the table owner (PostgreSQL 16+ membership WITH INHERIT FALSE) no longer leaves EXECUTE with PUBLIC. The refresh used to hand the function to the table owner first, after which that role could no longer revoke or grant on it, and PostgreSQL only warned. It now resets EXECUTE after the transfer, acting as the new owner when it cannot as itself, so exactly the roles holding TRUNCATE on lease_claim_batches can execute it, the former owner included. Migrations run with the owner as the effective role, as documented, were not affected.

Reads during prune

• admin::get_job and the other queue-storage point reads of one job no longer return JobNotFound for a failed job while queue prune moves its row out of the slot being pruned. Since 0.6.14 the read used one snapshot, and a prune that committed after that snapshot was taken left the old slot empty and the moved row invisible to it. Prune moves each retained failed row once per ring cycle, and on a new schema prunes slot 0 right after the first rotation, so a read polling for failed could hit it within a second of a worker starting. A read that finds nothing now reads once more in a new snapshot.

Changed

• While a backlog remains, each rescue pass holds the maintenance leader for up to rescue_pass_budget plus one batch, so promotion, rotation and prune wait up to that long between passes. On DLQ-enabled queues, exhausted attempts move to the DLQ at the drain rate, and Rescued lifecycle hooks that run slower than the drain have events dropped, with a warning, once 1,024 are outstanding on a worker.
• When the batch function's bounded retries run out, it makes one last attempt with ordinary lock waits, as 0.6.15 did. A per-row transaction that holds a lane for longer than about 2.5 s before it inserts one of the batch's keys can therefore still deadlock with it, as can cycles of three or more transactions. A cycle through a lock the batch's own transaction took in an earlier statement, which rolling back the batch cannot release, now ends in 40P01 after about 3 to 4 s instead of 1 to 2 s, and usually aborts the batch's transaction rather than the per-row one. A batch whose unique claims take longer than 500 ms to insert behaves as in 0.6.15; split such batches if they share keys or lanes with per-row producers.

Added

• ClientBuilder::rescue_pass_budget and rescue_pass_max_rows (Python rescue_pass_budget_ms and rescue_pass_max_rows) control how long and how far one queue-storage rescue pass drains. awa.maintenance.rescue.pass_size (by awa.rescue.kind and awa.rescue.pass_end) and awa.maintenance.rescue.oldest_attempt_age show a drain in progress.
• Durable operator halt and resume for maintenance.prune, in drain mode: awa control halt, status and resume, or the awa.halt_runtime_scope and awa.resume_runtime_scope functions from an existing SQL session. Compatible leaders observe it on their existing session about once a second and stop admitting rotation, prune, rollups, cleanup, admin metadata refresh and descriptor publication; enqueue, dispatch, handlers, cron, promotion, rescue and admin batches continue. Resume names the current halt's UUID. Status lists live runtimes without supported, recent reports and cannot report the fleet halted while any remain. Older binaries ignore the control. See runtime controls.

Full changelog: v0.6.15...v0.6.16