queue: actuation-lock contention burns the retry budget and kills the job — one-shot reopen enforcement lost in production (PR #9450)

[JSONbored]

Parent: #9461

Summary

When the per-PR actuation lock is contended, the contending job retries on a flat 5-second delay — but each retry counts against maxRetries, so the job dies after roughly 25 seconds of contention, while the lock it is waiting on is designed to be held for minutes. For the reopen-reclose policy this loses the enforcement permanently, because that path has exactly one trigger and no reconciler.

This is not theoretical: it happened in production on PR #9450.

Production evidence (edge-nl-01, 7 days to 2026-07-27)

Three selfhost_job_dead events, all lock contention:

{"level":"error","event":"selfhost_job_dead","attempts":N,
 "error":"pr actuation lock contended for JSONbored/loopover#9450 during reopen-reclose"}
{"level":"error","event":"selfhost_job_dead","attempts":N,
 "error":"pr actuation lock contended for JSONbored/loopover#9447 during public-surface-publish"}
{"level":"error","event":"selfhost_job_dead","attempts":N,
 "error":"pr actuation lock contended for JSONbored/loopover#9457 during public-surface-publish"}

The dead-letter queue is otherwise empty, so lock contention is currently the only thing killing jobs on this deployment.

Mechanism (verified at HEAD, 776c414)

src/queue/transient-locks.ts:168-176 — contention raises a retryable error with a flat 5 s delay:

export class PrActuationLockContendedError extends RetryableJobError {
  constructor(repoFullName: string, prNumber: number, policy: string) {
    super(`pr actuation lock contended for ${repoFullName}#${prNumber} during ${policy}`, {
      retryAfterMs: 5_000,
      retryKind: "pr_actuation_lock_contended",
    });

src/selfhost/pg-queue.ts:1628 — every retry increments attempts, and the job dies at the ceiling (maxRetries defaults to 5, :301):

if (attempts >= maxRetries) {
  await pool.query(`UPDATE ${TABLE} SET status='dead', ...`);

5 attempts × 5 s ≈ 25 seconds of tolerance. But PR_ACTUATION_LOCK_TTL_SECONDS = 600 (transient-locks.ts:139), and the lock legitimately wraps a full publish → AI-review → maintain unit. The queue already has the right primitive for this shape: a GitHub rate-limit failure is deferred without charging an attempt (pg-queue.ts:1585-1612, the row stays pending with deferred_by='rate_limit'). Lock contention gets no such treatment.

Why the reopen-reclose case is unrecoverable

maybeRecloseDisallowedReopen has exactly one call site, gated on the webhook action:

src/queue/processors.ts:6924

const reopenOutcome: ReopenRecloseOutcome =
  payload.action === "reopened" && installationId
    ? await maybeRecloseDisallowedReopen(env, deliveryId, installationId, repoFullName, pr, payload)
    : "allowed";

Nothing re-derives it. There is no sweep, watchdog or reconciler that asks "was this PR reopened by a non-maintainer after a bot/maintainer close?" — every other repair scan keys on surface staleness, missing dispositions, or pending closures. Once the job dies, the disallowed reopen stands, and the PR proceeds to a fresh review that the one-shot-close policy explicitly forbids.

The public-surface-publish deaths are less severe — the stale-surface repair sweep eventually re-drives them — but they still represent lost work and delayed verdicts.

Requirements

1. Lock contention must not consume a job's retry budget; it is a "come back later" condition, not a failure.
2. The reopen-reclose enforcement must survive a lost job, not depend on a single webhook delivery.
3. Contention must remain observable (it is a useful signal about lock hold times — see locks: 600s actuation TTL underruns the AI review it wraps since #9013, with no renewal → concurrent double actuation #9467 for the TTL/renewal side of this).

Deliverables

• Give PrActuationLockContendedError attempt-free deferral in both queue backends (pg-queue.ts, sqlite-queue.ts), mirroring the existing deferred_by='rate_limit' path: reschedule without incrementing attempts, with a bounded total deferral window so a genuinely wedged lock still surfaces rather than looping forever.
• Add jitter to the 5 s retry so several contenders do not re-collide in lockstep.
• Add a reconciler for disallowed reopens: a bounded scan for open PRs whose last close was by the bot/maintainer and which were reopened by a non-maintainer without a subsequent valid disposition. Ride an existing sweep tick, following the pattern of reconcileSurfaceWithoutDisposition (src/review/surface-disposition-reconciler.ts) — bounded lookback, capped per run, best-effort, re-enqueues rather than acting directly.
• Emit a metric for deferred-due-to-contention, separate from dead jobs, so the two are distinguishable in Grafana.
• Alert on any selfhost_job_dead whose retryKind is pr_actuation_lock_contended — after this fix it should be ~zero, so a recurrence means the deferral window is wrong.

Tests (must fail against current main)

• A job that hits lock contention N > maxRetries times is still pending (not dead), with attempts unchanged, until the bounded deferral window elapses.
• Both backends behave identically (parity test).
• The reopen reconciler detects a PR reopened by a non-maintainer after a bot close with no subsequent disposition, and enqueues re-enforcement.
• The reconciler does not fire for self-closes reopened by their author, or for the bot's own nightly re-review reopens (the existing exemptions in maybeRecloseDisallowedReopen).

Expected outcome

Actuation-lock contention becomes a scheduling delay rather than a lost mutation, and one-shot reopen enforcement no longer depends on a single webhook delivery surviving a 25-second race.

Follow-up

PR #9450 should be checked by hand — per the log above, its reopen enforcement was dropped.

[linear-code]

JSO-6528