fix(heartbeat): durably classify process_lost reaps for the BLO-12292 split (BLO-16181)

[kkroo]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work; agents run as external-lifecycle k8s Jobs (claude_k8s / opencode_k8s).
• The heartbeat reaper (reapOrphanedRuns) finalizes runs whose backing Job died or vanished by minting errorCode="process_lost".
• The BLO-16181 diagnostic spike found the mint stamped only a human-readable string, discarding whether a Job was ever created and what state it reached.
• That erased the single most useful discriminator for the fleet-wide process_lost instability (BLO-12292) and for reattach feasibility (BLO-12564): a run whose Job never got a name is unreattachable, while a run whose Job ran and vanished may be.
• This pull request records a stable classification (plus the raw signals it derives from) on resultJson.processLoss at mint time, from state the reap loop already holds — no extra kube round-trip.
• The benefit is that the BLO-16184 monitor can GROUP BY the classification to finish the root-cause split, and BLO-12564 can tell reattachable losses from unrecoverable ones.

Linked Issues or Issue Description

Board issue: BLO-16181 (implements the durable-capture recommendation from the BLO-16181 diagnostic spike). Parent: BLO-12292. Consumers: BLO-16184 (monitor), BLO-12564 (reattach). No GitHub issue mirror exists, so the underlying problem is described inline below following the bug template.

What happened

The reaper's process_lost mint discarded all backing-Job state, leaving no way to tell a Job-never-created loss from a Job-ran-then-vanished loss.

Expected behavior

Every process_lost mint durably records a stable classification + the signals it derives from, queryable at result_json->'processLoss'->>'classification'.

Steps to reproduce

1. An external-lifecycle agent Job is reaped as process_lost (pre-adapter setup death, or a started run whose Job left the live snapshot while silent).
2. Inspect heartbeat_runs.result_json — before this change there is no structured record of Job/name state.

Paperclip version

master @ this PR

Deployment mode

k8s external-lifecycle adapters (claude_k8s / opencode_k8s)

What Changed

• New pure module server/src/services/process-loss-classification.ts (classifyProcessLoss + buildProcessLossCapture), exhaustively unit-tested with no DB.
• The reaper wires the capture into the process_lost mint and threads the pre-adapter Job liveness it already resolves; zero new kube calls — every signal is already in hand at the mint.
• Local (non-external) reaps get a minimal {externalLifecycleRun:false, classification:"local"} marker so the monitor can exclude them without a join.
• Pod-level terminated.{reason,exitCode} is intentionally not captured here: runs reaching this mint have no inspectable pod, and a confirmed exact-name 404 finalizes upstream as job_missing, never process_lost.

Verification

• cd server && npx vitest run src/__tests__/process-loss-classification.test.ts — pure unit tests (general shard, no DB), every classification branch. 10/10 green locally.
• heartbeat-process-recovery.test.ts — 4 DB-backed reaper tests (serialized shard, run in CI) assert the mint persists the correct processLoss block for pre-adapter-unstamped, started-job-absent (unstamped + stamped), and pre-adapter-stamped-dead (liveness threading).
• cd server && npx tsc --noEmit -p tsconfig.json — clean.

Risks

Low risk. The change is additive: it only adds a processLoss key to resultJson at the process_lost mint and threads a new local let into the reap loop; no control flow, errorCode, or existing field changes. mergeHeartbeatRunStopMetadata preserves the injected key (spreads resultJson first). Existing reaper tests assert resultJson via toMatchObject (subset), so the added key does not break them.

Model Used

Claude Opus 4.8 (model ID claude-opus-4-8), 1M context window, extended thinking, with tool use / code execution.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I searched the GitHub PR list (open + recently closed) for similar/duplicate PRs and confirmed this is not a duplicate
• I have either (a) linked existing issues OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass (pure tests + tsc; DB tests run in CI)
• I have added or updated tests where applicable
• I have considered and documented any risks above

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-16181
🔗 Paperclip issue: BLO-16184
🔗 Paperclip issue: BLO-12292
🔗 Paperclip issue: BLO-12564

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

🔗 Paperclip issue: BLO-16181
🔗 Paperclip issue: BLO-16184
🔗 Paperclip issue: BLO-12292
🔗 Paperclip issue: BLO-12564

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip