fix(heartbeat): hold process_lost during cold-boot kube-blind window so live Jobs reattach across restart (BLO-12564)

[kkroo]

Thinking Path

BLO-12564 asks the worker to reconcile/reattach in-flight runs across a server restart instead of zeroing them to process_lost. An understand-phase fan-out over the reaper, the run↔Job liveness/linkage, the process_lost side-effects, the monitor loop, and the test harness showed the issue's premise is largely stale. An alive-Job run already reattaches on the kube-up path (BLO-13176 plus the 2026-05-23 live-but-quiet guards), the run↔Job linkage (externalRunId) already survives a restart, and the "priority auto-downgraded to low" behavior does not exist in code at all — escalateStrandedAssignedIssue writes only {status: blocked, blockedByIssueIds}, never a priority. The instrumentation from the rest of the BLO-12292 tree (#675, #677, #681, #683) confirms this split. The one genuine residual is the cold-boot window: on a fresh worker boot the in-cluster kube client may not be serving yet, so the startup reap (which runs before the periodic timer in server/src/index.ts) and the first ticks can run fully kube-blind — both listAgentJobRunStatuses() and listLiveAgentJobRunIds() return null — and an external-lifecycle run whose agent Job is actually still Running resolves liveness "unknown" and gets minted process_lost past the silence floor. That is exactly the server restart the ticket names, in its worst form. This PR closes that window with a bounded, worker-opt-in grace and pins the whole reattach contract with regression tests.

What Changed

Adds a cold-boot reattach guard in reapOrphanedRuns (server/src/services/heartbeat.ts): while the worker has been up for less than coldBootReattachGraceMs and the reaper is fully kube-blind (liveJobRunIds === null), an external-lifecycle run that would otherwise be minted process_lost is left running, so the next warm tick reattaches a still-Running Job (or reaps a genuinely-dead one). It is placed at the single mint convergence point and is symmetric to the BLO-12563 shutdown-side drain guard. The grace defaults to 0 (off) on the factory and is wired to 5 min only on the long-lived worker (server/src/index.ts); every per-request heartbeatService() construction and the whole test suite keep the steady-state null-liveness reap contract unchanged. Because liveJobRunIds === null structurally implies jobRunStatuses === null, the guard can never intercept a kube-confirmed finalize (job_missing / job_failed / external_lifecycle_stale_killed) — only the blind silence-floor mint.

What happened

On a paperclip-0 worker restart, the startup reap could run before the in-cluster kube client was ready, see both liveness lists return null, and mint process_lost on external-lifecycle runs whose agent Job (owned by its own k8s controller) was still Running — zeroing a run that should simply have been re-adopted.

Expected behavior

A worker restart with an in-flight run whose agent Job is still Running leaves the run running (reattach on the next warm tick), not a process_lost recovery transition; runs with genuinely no live Job still recover after the silence floor; priority is never auto-downgraded.

Steps to reproduce

Restart the worker while an external-lifecycle run is in flight and the in-cluster kube API is briefly unreachable at boot; before this change the first reaper tick mints process_lost on the still-Running Job.

Verification

pnpm --filter @paperclipai/server typecheck passes clean. Added 4 regression tests in heartbeat-process-recovery.test.ts (Linux-CI-only; the suite skips on macOS embedded-postgres): within-grace shields plus past-grace still-reaps, for both the started and pre-adapter paths (BLO-12564: within the cold-boot grace ... / ... past the cold-boot grace ...). Each mirrors an existing steady-state sibling's seed (the "reaps ... silent past the staleness window" started case and the BLO-16183 pre-adapter aging) so the "reaches the mint absent the guard" premise is proven by parity. Two independent adversarial reviews found no defect across correctness and test-path items.

Risks

Low. The guard fires only in the first grace ms of worker uptime and only when kube is fully unavailable (a transient blind tick self-heals on the next tick); the uptime bound is monotonic in real time, so there is no permanent shield, and every degenerate input (NaN/undefined) fails safe toward reaping. Worst case is a genuinely-orphaned pre-adapter run recovering up to 5 min later during a boot-time kube outage — bounded, and BLO-16182 already reclassifies process_lost as transient_infra with a retry budget, so that harm was already blunted. No change to local-adapter runs, to the kube-up reattach path, or to the steady-state null-liveness reap (pinned by the pre-existing tests, which construct the reaper with the grace off).

Model Used

Claude Opus 4.8 (1M context)

---

• I searched the GitHub PR list (open + recently closed) for similar PRs and confirmed this is not a duplicate.

Linked issue: BLO-12564 — code sibling of the BLO-12292 process_lost tree (instrumentation landed in #675, #677, #681, #683).

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-16183
🔗 Paperclip issue: BLO-12563
🔗 Paperclip issue: BLO-13176
🔗 Paperclip issue: BLO-12292
🔗 Paperclip issue: BLO-12564
🔗 Paperclip issue: BLO-16182

[allyblockcast]

🔗 Paperclip issue: BLO-16183
🔗 Paperclip issue: BLO-12563
🔗 Paperclip issue: BLO-13176
🔗 Paperclip issue: BLO-12292
🔗 Paperclip issue: BLO-12564
🔗 Paperclip issue: BLO-16182

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

@ally Please perform the pre-landing structural review for BLO-12564, focusing on the cold-boot grace boundary in reapOrphanedRuns: verify the guard only covers fully kube-blind process_lost minting, cannot suppress kube-confirmed finalization, defaults off outside the long-lived worker, and that the four regression tests prove both within-grace shielding and past-grace recovery paths.