queue: no 'current focus' concept exists, and oldest-first ordering is defeated by 5 of 8 regate producers omitting prCreatedAt

[JSONbored]

Summary

The maintainer's expectation is that ORB works through PRs one at a time, oldest to newest. It does not, and there is no mechanism that could — no "current focus" concept exists anywhere in the codebase. The one real ordering mechanism that does exist is silently defeated by five of its eight producers.

Current state (verified at HEAD, 776c414)

No focus concept exists. The only mutex is per-PR — claimPrActuationLock(env, repoFullName, prNumber) (src/queue/transient-locks.ts:143-153). There is no repo-level or global serialization key; production Redis has zero keys matching *focus*. Queue concurrency is 8 (src/selfhost/pg-queue.ts:306-308), so up to eight PRs are actively processed at once.

The merge train is not it. src/review/merge-train.ts constrains merging only — checked at src/services/agent-action-executor.ts:643 behind if (action.actionClass === "merge" && ctx.mergeTrainMode !== "off"). It does not gate processing and does not gate update_branch. It is also overlap-scoped, not FIFO (merge-train.ts:130): an unrelated older sibling never blocks a newer PR.

Worth checking separately: on current main, mergeTrainMode is hardcoded "off" in the settings row mapper (src/db/repositories.ts:914) with no manifest override anywhere in src/ — i.e. the train appears to be dead code on HEAD. Production (orb-v3.5.0-beta.10) logged 17 merge_train_blocked events, so the deployed build predates that change. If the train is meant to be live, that is a separate regression.

Ordering exists but is defeated. jobClaimSortKey (src/selfhost/queue-common.ts:837-846) sorts agent-regate-pr rows by PR createdAt ascending, consumed by ORDER BY priority DESC, claim_sort_key, run_after, id (pg-queue.ts:1286). That is genuine oldest-first, and JobMessage.prCreatedAt's own doc (src/types.ts:43) says it exists so queues "drain contributor PR work oldest-first". Three things break it:

1. github-webhook jobs are priority 10 vs regate's 9 — any webhook preempts the ordered work.
2. Concurrency 8 claims eight rows at once, so "first" is eight-wide.
3. Five of the eight regate producers omit prCreatedAt entirely, falling back to LEGACY_AGENT_REGATE_SORT_BASE_MS + prNumber ≈ 9.5e11 — which sorts ahead of every real 2026 PR (≈1.78e12). Offenders: processors.ts:5032 (trailing mergeable-unknown), processors.ts:5086 (over-cap sibling wake), src/review/pr-reconciliation.ts:132, src/review/surface-disposition-reconciler.ts:81, src/api/routes.ts:4933 (manual, priority 99 — intentional).

Deliverables

Phase 1 — make the existing ordering actually work (low risk, do first)

• Thread prCreatedAt through the four unintentional omissions (processors.ts:5032, processors.ts:5086, pr-reconciliation.ts:132, surface-disposition-reconciler.ts:81). Leave the manual route as-is; document why.
• Add a coalesce key to maybeEnqueueSiblingRegateForMergedPr (processors.ts:1402-1437) — it is the only wake producer in that file without one, so three merges six seconds apart currently fan out 45 jobs for the same 15 siblings.
• Re-key isRegateRepairExhausted's budget on repo#pr instead of repo#pr#headSha (processors.ts:1121-1123). A successful rebase changes the head SHA and resets the 5-attempt budget to zero — the identical bug already fixed for the fresh-rebase counter at :4787-4790, which the comment there explains at length.
• Add a lint or type-level guard so a new agent-regate-pr producer cannot omit prCreatedAt silently.

Phase 2 — an explicit focus gate, if still wanted after Phase 1

Gate at actuation, never at job admission. Admission-gating would break the readiness/CI-wait machinery, starve the surface publish contributors depend on, and stall the repair and reconciliation paths.

• Add a step 8c in executeAgentMaintenanceActions mirroring the existing merge-train gate (agent-action-executor.ts:643-676) exactly: same listOtherOpenPullRequests fetch, same off | audit | enforce dial, same wedge-audit and alerting wiring — applied to actionClass === "update_branch" as well as merge.
• Reuse merge-train.ts's own isOlder (createdAt, falling back to PR number, :114-124) and all of its eviction filters. The 24h MERGE_TRAIN_MAX_WAIT_MS eviction (:59, :131-135) and the orb(merge-train): a blocked head-of-line PR stalls every PR behind it — 5 PRs blocked 4h behind one manual-review label #9039 manual-review eviction (:128) exist because of the confirmed fix(test): add required default field to queue-3 pause commandAuthorization fixture #8735 incident (57 denials across 5 PRs over 4 hours); a blanket FIFO without them reintroduces it.
• Ship in audit mode first and compare predicted vs actual orderings before enforcing.

Known interaction to handle: sweepRepoRegate stamps last_regated_at at dispatch (processors.ts:1567-1580), so a PR that loses the focus race and defers would burn its one sweep-eligible regate under #never-endless-reregate (src/settings/agent-sweep.ts:161-174) and never be re-examined. Either move the stamp to completion, or give deferred PRs repair-priority treatment.

Trade-off to decide explicitly

Strict global oldest-first is a behaviour change, not a tightening. The merge train is overlap-scoped precisely so an unrelated older PR does not block a newer one; global FIFO means a slow PR at the head delays everything behind it, bounded only by the 24h eviction. Phase 1 may deliver enough of the desired ordering without that cost — worth measuring before committing to Phase 2.

[linear-code]

JSO-6563

[JSONbored]

Correction while implementing: the claim that maybeEnqueueSiblingRegateForMergedPr has no coalesce key — and therefore that "three merges six seconds apart fan out 45 jobs for the same 15 siblings" — is wrong.

agent-regate-pr jobs already coalesce on agent-regate-pr:${repo}#${pr} (src/selfhost/queue-common.ts, in the job_key derivation). Two wakes for the same sibling merge into one pending row.

The audit conflated two different things: the queue coalesce key (which exists, repo+PR scoped) and the per-producer cooldown keys that the neighbouring wake producers carry in the transient cache. It is true that the sibling-merge wake has no cooldown, but that is a much weaker statement than "no coalescing" and does not produce the 45-job fan-out described.

Nothing was changed for this, and PR #9511 does not claim to have fixed it.

The rest of Phase 1 is implemented in #9511:

• prCreatedAt threaded through the open-PR reconciler, the surface-disposition reconciler, and the contributor-cap wake. Without it jobClaimSortKey falls back to LEGACY_AGENT_REGATE_SORT_BASE_MS + prNumber (~9.5e11), which sorts ahead of every real 2026 PR (~1.78e12) — so these silently preempted older contributor work.
• isRegateRepairExhausted re-keyed from repo#pr#headSha to repo#pr. A successful rebase minted a new SHA and reset the 5-attempt budget to zero, which is the same bug MAX_FRESH_REBASE_FORCES already fixed for the fresh-rebase counter. Note this resets existing budgets once on deploy, since historical audit rows carry the old target key.

Phase 2 (the explicit focus gate) is deliberately still open here. It is a behaviour change rather than a tightening — strict global FIFO means one slow PR delays everything behind it, which is exactly why the merge train was built overlap-scoped — and it is worth measuring after Phase 1 and #9497 land, since #9497 alone removes the burst behaviour that motivated it.