fix(heartbeat): cap external-lifecycle dispatch to one run (BLO-13176 follow-on)

[kkroo]

Summary

Follow-on to #576. With the pre-adapter reaper fixed, BLO-12825 still could not complete — its runs kept dying "before external adapter invocation", but not from the reaper. Root cause: over-dispatch.

An external-lifecycle (k8s Job) agent can only run one Job at a time — the runningCount>0 and hasActiveJobForAgent gates in startNextQueuedRunForAgent reject any second dispatch while one is active. But on an idle agent (runningCount 0), availableSlots = maxConcurrentRuns (e.g. 3), so the claim loop claimed and executeRun'd up to 3 queued runs (one per distinct issue) concurrently. Only the first to reach Job creation won the single slot; the losers sat pre-adapter with no Job and were (correctly) reaped as process_lost. The race is first-lease-wins, not priority-ordered.

Confirmed in prod (2026-07-02)

• BLO-12825's critical run 49ca8e50 leased 17:32:11, never got a k8s Job, reaped 17:36:17 "before external adapter invocation".
• A normal-priority sibling (b1c7a483) that leased 17:32:06 — 5 s earlier — got the Job and ran fine (8 min+).

So a critical issue kept losing the single slot to whatever leased a few seconds sooner.

Fix

Cap external-lifecycle dispatch to a single run:

const effectiveMaxConcurrentRuns = hasExternalLifecycle(agent.adapterType)
  ? 1
  : policy.maxConcurrentRuns;
const availableSlots = Math.max(0, effectiveMaxConcurrentRuns - runningCount);

This (a) stops leasing work we can't immediately give a Job — no more doomed surplus pre-adapter runs — and (b) gives the one slot to the top of the existing priority sort, so a critical issue wins instead of the fastest leaser. Local (child-process) adapters are unaffected and keep full concurrency. The steady-state runningCount>0 / hasActiveJobForAgent gates already enforced effective-concurrency-1 for subsequent dispatches; this just makes the initial idle-agent burst consistent with them.

Test

New case in heartbeat-process-recovery.test.ts: an idle opencode_k8s agent with maxConcurrentRuns=3 and three queued distinct-issue runs (critical is the newest, so the createdAt tie-break / first-lease-wins would not pick it) claims exactly one run — and it's the critical one; the other two stay queued.

Full heartbeat-process-recovery + heartbeat-dispatch-priority-sort suites green: 128/128. Server typecheck clean.

🤖 Generated with Claude Code

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (degraded — nested CLI unavailable in this k8s runtime; reasoning applied directly against /tmp/pr.diff).

Looks good — small, well-scoped fix with strong test coverage. No Critical or Important findings.

Suggestions (2)

• [native-codex] .github PR-template quality gate — the commitperclip bot flagged this PR as missing required description sections (## Thinking Path, ## What Changed, ## Verification, ## Risks, ## Model Used) and the dedup-search checkbox. The review CI check is currently failing because of this, independent of the code change itself — worth fixing before merge so the gate goes green.
• [pr-review-toolkit:comment-analyzer] server/src/services/heartbeat.ts:11341-11354 — the added comment is long (14 lines) but earns its length: it documents a genuinely non-obvious race (first-lease-wins vs. priority-ordered dispatch) and the two invariants the existing gates already provide. No change requested, just flagging it was reviewed rather than skipped.

Strengths

• The fix is minimal and precisely targeted: effectiveMaxConcurrentRuns only changes behavior for hasExternalLifecycle adapters, leaving local child-process adapters at full concurrency.
• Verified against the surrounding code (server/src/services/heartbeat.ts:11403-11467) that prioritizedRuns is priority-sorted before the availableSlots cap is applied in the claim loop, so capping to 1 correctly hands the single slot to the top of the existing priority sort rather than an arbitrary run — this matches the PR's stated intent.
• New test (heartbeat-process-recovery.test.ts:1839-1961) is a good regression test: it seeds three distinct-issue queued runs with the critical one seeded newest specifically to prove the priority sort — not the createdAt tie-break — is what wins the slot, and asserts both claimed.length === 1 and the two losers stay queued rather than being claimed-and-doomed.
• normalizeMaxConcurrentRuns clamps to HEARTBEAT_POLICY_MAX_CONCURRENT_MIN, so there's no risk of this new cap silently overriding an intentional "0 = disabled" policy — confirmed there's no such value in the valid range.
• CI is green except the unrelated PR-template gate above (Build, both test suites, typecheck, e2e, security-review, policy all pass).

Recommended Action

1. Fill in the missing PR description sections to unblock the review CI gate.
2. Merge — no code-level blockers.