fix(heartbeat): stop reaper from killing live pre-adapter runs (BLO-13176)

[kkroo]

Summary

The executor's orphaned-heartbeat reaper was force-killing live agent runs before they ever invoked the model adapter, surfacing fleet-wide as process_lost ("k8s job terminated … before external adapter invocation"). This is the confirmed blocker for BLO-12825 (critical — could not complete a single run for days) and it lines up with the dashboard's chronic run-failure counts.

Follow-on to BLO-12996 / #575 (which added reapOrphanedRuns force-reap of live-but-silent Jobs). BLO-12996 correctly reaps truly-dead Jobs — but the reaper was also killing live-but-pre-adapter runs whose setup exceeded the grace window.

Evidence (executor's own logs, paperclip-0, 2026-07-02)

• BLO-12825 run 6d4843b2 reaped at 08:58:17 (reaped orphaned heartbeat runs), ~5 min after preRun, with a 2/2 Running pod.
• Penstock proxy+vault logs: zero model relays in the death window — no model request was ever made.
• Agent Job spec: activeDeadlineSeconds: null, restartPolicy: Never — the ~5-min death is the executor deleting the Job, not a k8s time cap.

Two defects, both fixed

1. Over-reaping pre-adapter runs. Once the 5-min pre-adapter grace expired, a run with no adapter.invoke event fell straight through to the process_lost finalize without ever consulting kube Job liveness — so a pod still provisioning (image pull, repo clone, opencode/claude cold boot) past 5 min was killed while alive. Now a pre-adapter run past the grace consults the kube signal the reaper already gathered this pass: only positive evidence of a live Job protects it (with a 45-min hard-stale escape hatch for a genuinely wedged setup). dead/unknown keep the pre-existing fall-through, so real orphans and the kube-unavailable degraded path reap exactly as before. Mirrors the 2026-05-23 guard that already protects live-but-quiet started runs.

2. deletedJobs:0 force-kill thrash. The hard-stale force-kill deleted the Job before claiming the run terminally, so overlapping reaper passes could both select the still-running row and both delete + warn (observed: 5 consecutive force-killed live-but-silent … deletedJobs:0 warns for one runId). The staleKill path now compare-and-swaps status=running → failed first (setRunStatusIfRunning); only the winner deletes the Job, losers no-op. A failed delete is reaped next pass by cleanupTerminalExternalLifecycleJobs.

Tests

4 new cases in heartbeat-process-recovery.test.ts:

• pre-adapter run with a live Job (kube status snapshot) → not reaped;
• pre-adapter run alive in the kube list snapshot → not reaped;
• pre-adapter run whose Job is confirmed gone by name → still reaped (guards against over-protection / BLO-8827 immortal orphans);
• pre-adapter run wedged past the 45-min hard ceiling with a live Job → force-killed, and a second pass does not re-fire (idempotency).

Full suite green: 125/125, including the three BLO-12996 hard-stale guards and the 2026-05-23 live-but-quiet guard. Server typecheck clean.

🤖 Generated with Claude Code

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Looks good. I traced the full reapOrphanedRuns control flow (not just the diff) against the current file at head c51732e2 to check the new pre-adapter liveness branch interacts correctly with the pre-existing STARTED-run liveness branch below it, and it's internally consistent:

• resolveExternalLifecycleJobLiveness correctly falls back to "unknown" when both jobRunStatuses and liveJobRunIds are null (kube fully unavailable), which preserves the pre-existing silence-floor fallthrough — matches the PR's stated intent that only "dead"/"unknown" reap as before.
• The CAS-first finalize (setRunStatusIfRunning) correctly gates the destructive deleteAgentJobsForRun call behind winning the running → failed transition, and reuses an existing helper (already used elsewhere for the STARTED hard-stale guard) rather than inventing new locking.
• Verified the 45-min hard-stale math against the test fixtures: seedRunFixture pins startedAt/createdAt to a fixed historical date but lets lastOutputAt default to real wall-clock new Date(), and externalLifecycleRecentRefTime takes the max — so the 6-min and 50-min test cases correctly land on either side of EXTERNAL_LIFECYCLE_HARD_STALE_MS against the real now the reaper uses. No off-by-fixture-date bug.
• The "STILL reaps... confirmed gone by name" test correctly exercises the one path where a pre-adapter run remains genuinely reapable (BLO-8827 regression guard).

Suggestions (2)

• [code] server/src/services/heartbeat.ts (finalizeExternalLifecycleTerminalRun appendRunEvent payload, ~line 10327) — the force-kill event payload for staleKill runs includes jobPhase/jobReason/jobMessage but no externalLifecyclePreAdapter marker, unlike the plain process_lost path a few lines down which does stamp it (line ~10896). An operator triaging external_lifecycle_stale_killed events can't tell "wedged before ever invoking the adapter" apart from "wedged after invoking it" without cross-referencing hasAdapterInvocationEvent separately. Worth threading the same flag through for consistency.
• [perf] resolveExternalLifecycleJobLiveness issues an extra readAgentJobRunStatusByName kube call for every pre-adapter run that misses the snapshot, on every reaper tick until it resolves. This mirrors an existing pattern in the STARTED-run branch, so it's not a new class of cost, but worth keeping an eye on fleet-wide kube QPS if the pre-adapter population grows (e.g. an image-pull incident stalling many runs at once).

Strengths

• Root-caused with concrete evidence (executor logs, pod phase, penstock silence) rather than a speculative fix.
• 4 new tests cover exactly the four branch outcomes (live+protected, live via list-fallback, confirmed-dead still reaps, wedged past hard ceiling + idempotent second pass) — the idempotency test is a good catch given the deletedJobs:0 thrash this PR also fixes.
• The CAS fix for the second defect (deletedJobs:0 thrash) is a real bug and the ordering fix (claim before delete) is the correct direction.

Recommended Action

Nothing blocking. Consider the payload-marker suggestion opportunistically.