Summary
The per-PR actuation lock and the AI-review lock are mis-sized on both sides simultaneously, and their lifecycle has three additional edges where a lock is freed while its owner is still running. Individually these are separate bugs; together they mean the engine's central mutual-exclusion primitive cannot be relied on to actually exclude.
Found by an adversarial audit on 2026-07-27 against origin/main @ 776c4147a, with the contender-side failure confirmed in production on edge-nl-01.
The shape of the problem
| Side |
Defect |
Consequence |
| Contender |
Contention retries charge maxRetries; job dies after ~25 s |
Lost mutation — confirmed live (#9465) |
| Holder |
600 s TTL is shorter than the work it now wraps; no renewal exists |
Lock expires mid-work → double actuation |
| Lifecycle |
Boot flush, SIGTERM release, and registry keying each free a live holder's lock |
Double actuation without any expiry |
The contender side is filed separately as #9465 because it is independently actionable and production-confirmed. This epic covers the holder and lifecycle sides.
Why one epic
A single primitive fixes most of it: a token-checked lock renewal heartbeat. The queue already proved this pattern for its own processing lease in #9023 (heartbeatProcessingLease, src/selfhost/pg-queue.ts:1323). Applying the same shape to claimTransientLock holders resolves the two TTL-underrun defects outright and narrows the SIGTERM window.
Deployment context (verified live, 2026-07-27)
- Single-instance deployment today:
loopover-loopover-1, RestartCount=0, Postgres queue backend, Redis at 1.64 MB / 256 MB.
maxmemory-policy allkeys-lru — the lock keys (pr-actuation-lock:*, ai-review-lock:*, contributor-cap-lock:*) live in an LRU-evictable keyspace alongside caches. Usage is 0.6% of the cap today, so this is latent, but under memory pressure Redis may evict a held lock, silently defeating exclusivity with no expiry and no log.- Several lifecycle defects below are latent only because the deployment is single-instance.
docker-compose.yml:14 advertises "multi-instance capable" and the pg queue backend exists precisely to support it — the moment topology changes, they become live.
Expected outcome
A lock, once acquired, is held for exactly as long as its owner is doing the work it guards — no longer (no strand) and no shorter (no silent expiry). No code path frees a lock whose owner is still running.
Acceptance criteria
Related
Summary
The per-PR actuation lock and the AI-review lock are mis-sized on both sides simultaneously, and their lifecycle has three additional edges where a lock is freed while its owner is still running. Individually these are separate bugs; together they mean the engine's central mutual-exclusion primitive cannot be relied on to actually exclude.
Found by an adversarial audit on 2026-07-27 against
origin/main@776c4147a, with the contender-side failure confirmed in production onedge-nl-01.The shape of the problem
maxRetries; job dies after ~25 sThe contender side is filed separately as #9465 because it is independently actionable and production-confirmed. This epic covers the holder and lifecycle sides.
Why one epic
A single primitive fixes most of it: a token-checked lock renewal heartbeat. The queue already proved this pattern for its own processing lease in #9023 (
heartbeatProcessingLease,src/selfhost/pg-queue.ts:1323). Applying the same shape toclaimTransientLockholders resolves the two TTL-underrun defects outright and narrows the SIGTERM window.Deployment context (verified live, 2026-07-27)
loopover-loopover-1,RestartCount=0, Postgres queue backend, Redis at 1.64 MB / 256 MB.maxmemory-policy allkeys-lru— the lock keys (pr-actuation-lock:*,ai-review-lock:*,contributor-cap-lock:*) live in an LRU-evictable keyspace alongside caches. Usage is 0.6% of the cap today, so this is latent, but under memory pressure Redis may evict a held lock, silently defeating exclusivity with no expiry and no log.docker-compose.yml:14advertises "multi-instance capable" and the pg queue backend exists precisely to support it — the moment topology changes, they become live.Expected outcome
A lock, once acquired, is held for exactly as long as its owner is doing the work it guards — no longer (no strand) and no shorter (no silent expiry). No code path frees a lock whose owner is still running.
Acceptance criteria
transient-locks.ts's header, of what each lock guarantees under: TTL expiry, process kill, redeploy, Redis eviction, and multi-instance operation.Related