fix(heartbeat): reaper must ignore updatedAt churn for external-lifecycle runs (BLO-8827) - #287
Merged
Merged
Conversation
…ycle runs (BLO-8827) Live incident 2026-06-03: a MulticastEngineer opencode_k8s run sat `running` for 4h+ with no backing k8s Job and no adapter.invoke event (a pre-adapter orphan created at a pod rollout). The reaper was actively reaping other orphans, but kept skipping this one. Root cause: the reaper used heartbeat_runs.updated_at as a freshness proxy in two gates — the pre-adapter grace (externalLifecycleRecentRefTime includes updatedAt) and the periodic staleThreshold gate (now - updatedAt < 5min skips). But updated_at is a generic "row was touched" timestamp, and the silent-active-run / board-recovery review loop (the BLO-8912/8913 reviews opened for this very run) bumps it every ~minute. So a dead run under review looked "recently active" forever — the review meant to recover it was the thing shielding it from the reaper. Same anti-pattern as trusting process_pid. Fix: external-lifecycle staleness/grace keys on genuine activity only — lastOutputAt / startedAt / createdAt / finishedAt — never updatedAt. - externalLifecycleRecentRefTime: drop updatedAt from the max(). - staleThreshold gate: for external-lifecycle runs use externalLifecycleRecentRefTime; local adapters keep updatedAt (their liveness is pid/group-tracked, not Jobs). A healthy streaming agent is unaffected (fresh lastOutputAt keeps it out of the reaper); this only stops updatedAt churn from protecting dead runs, so it cannot cause false-positive reaping (the 2026-05-06/05-23 RCA concern). Test (embedded Postgres, TDD — watched it fail first): a pre-adapter opencode_k8s orphan with ancient startedAt but freshly-churned updatedAt and no Job is now reaped (process_lost); previously it was skipped indefinitely. Full heartbeat-process-recovery suite green (85), tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live incident (root cause)
2026-06-03: a MulticastEngineer
opencode_k8srun satrunningfor 4h+ with no backing k8s Job and noadapter.invokeevent — a pre-adapter orphan created at a pod rollout. The reaper was actively reaping other orphans (manyreaped orphaned heartbeat runslog lines) but kept skipping this one. The detection layer worked (BLO-8912"silent active run" +BLO-8913"Board recovery: cancel/finalize stale run 9852ea6a") — butBLO-8913wasblocked, so it never executed.Why the reaper skipped it: it used
heartbeat_runs.updated_atas a freshness proxy in two gates:externalLifecycleRecentRefTimeincludedupdatedAtin itsmax(...);staleThresholdMsgate —now - run.updatedAt < 5min → skip.updated_atis a generic "row was touched" timestamp, and the silent-active-run / board-recovery review loop (the BLO-8912/8913 reviews opened for this exact run) bumps it every ~minute (observed:updated_at12s old on a 4h14m run). So a dead run under review looked "recently active" forever — the review meant to recover it was the very thing shielding it from the reaper. Same anti-pattern as trustingprocess_pid(which is always NULL for these runs).Fix
External-lifecycle staleness/grace now keys on genuine activity only —
lastOutputAt/startedAt/createdAt/finishedAt— neverupdatedAt:externalLifecycleRecentRefTime: dropupdatedAtfrom themax().staleThresholdgate: for external-lifecycle runs useexternalLifecycleRecentRefTime; local adapters keepupdatedAt(their liveness is pid/group-tracked, not Job-tracked).A healthy streaming agent is unaffected — fresh
lastOutputAtkeeps it out of the reaper. This only stopsupdatedAtchurn from protecting dead runs, so it cannot cause false-positive reaping (the 2026-05-06 / 2026-05-23 RCA concern that hardened this path).Test (TDD — watched it fail first)
A pre-adapter
opencode_k8sorphan with ancientstartedAtbut freshly-churnedupdatedAtand no Job is now reaped (process_lost); previously it was skipped indefinitely. Fullheartbeat-process-recoverysuite green (85),tsc --noEmitclean.Relationship to #283
Sibling to #283 (image-bump starvation + Job-identity persistence). Same family: BLO-8746/BLO-8827 "reaper never reaps stuck
runningruns." #283 fixed the image-bump deadlock + persistedexternal_run_id; this fixes theupdatedAt-churn shield. On merge+deploy, the live 4h MulticastEngineer zombie should be reaped on the next reaper tick (will verify).🤖 Generated with Claude Code