Skip to content

fix(heartbeat): reaper must ignore updatedAt churn for external-lifecycle runs (BLO-8827) - #287

Merged
kkroo merged 1 commit into
masterfrom
omar/reaper-ignore-updatedat-churn
Jun 3, 2026
Merged

fix(heartbeat): reaper must ignore updatedAt churn for external-lifecycle runs (BLO-8827)#287
kkroo merged 1 commit into
masterfrom
omar/reaper-ignore-updatedat-churn

Conversation

@kkroo

@kkroo kkroo commented Jun 3, 2026

Copy link
Copy Markdown

Live incident (root cause)

2026-06-03: a MulticastEngineer opencode_k8s run sat running for 4h+ with no backing k8s Job and no adapter.invoke event — a pre-adapter orphan created at a pod rollout. The reaper was actively reaping other orphans (many reaped orphaned heartbeat runs log lines) but kept skipping this one. The detection layer worked (BLO-8912 "silent active run" + BLO-8913 "Board recovery: cancel/finalize stale run 9852ea6a") — but BLO-8913 was blocked, so it never executed.

Why the reaper skipped it: it used heartbeat_runs.updated_at as a freshness proxy in two gates:

  1. pre-adapter grace — externalLifecycleRecentRefTime included updatedAt in its max(...);
  2. the periodic staleThresholdMs gate — now - run.updatedAt < 5min → skip.

updated_at is a generic "row was touched" timestamp, and the silent-active-run / board-recovery review loop (the BLO-8912/8913 reviews opened for this exact run) bumps it every ~minute (observed: updated_at 12s old on a 4h14m run). So a dead run under review looked "recently active" forever — the review meant to recover it was the very thing shielding it from the reaper. Same anti-pattern as trusting process_pid (which is always NULL for these runs).

Fix

External-lifecycle staleness/grace now keys on genuine activity onlylastOutputAt / startedAt / createdAt / finishedAt — never updatedAt:

  • externalLifecycleRecentRefTime: drop updatedAt from the max().
  • staleThreshold gate: for external-lifecycle runs use externalLifecycleRecentRefTime; local adapters keep updatedAt (their liveness is pid/group-tracked, not Job-tracked).

A healthy streaming agent is unaffected — fresh lastOutputAt keeps it out of the reaper. This only stops updatedAt churn from protecting dead runs, so it cannot cause false-positive reaping (the 2026-05-06 / 2026-05-23 RCA concern that hardened this path).

Test (TDD — watched it fail first)

A pre-adapter opencode_k8s orphan with ancient startedAt but freshly-churned updatedAt and no Job is now reaped (process_lost); previously it was skipped indefinitely. Full heartbeat-process-recovery suite green (85), tsc --noEmit clean.

Relationship to #283

Sibling to #283 (image-bump starvation + Job-identity persistence). Same family: BLO-8746/BLO-8827 "reaper never reaps stuck running runs." #283 fixed the image-bump deadlock + persisted external_run_id; this fixes the updatedAt-churn shield. On merge+deploy, the live 4h MulticastEngineer zombie should be reaped on the next reaper tick (will verify).

🤖 Generated with Claude Code

…ycle runs (BLO-8827)

Live incident 2026-06-03: a MulticastEngineer opencode_k8s run sat `running`
for 4h+ with no backing k8s Job and no adapter.invoke event (a pre-adapter
orphan created at a pod rollout). The reaper was actively reaping other
orphans, but kept skipping this one.

Root cause: the reaper used heartbeat_runs.updated_at as a freshness proxy in
two gates — the pre-adapter grace (externalLifecycleRecentRefTime includes
updatedAt) and the periodic staleThreshold gate (now - updatedAt < 5min skips).
But updated_at is a generic "row was touched" timestamp, and the
silent-active-run / board-recovery review loop (the BLO-8912/8913 reviews
opened for this very run) bumps it every ~minute. So a dead run under review
looked "recently active" forever — the review meant to recover it was the thing
shielding it from the reaper. Same anti-pattern as trusting process_pid.

Fix: external-lifecycle staleness/grace keys on genuine activity only —
lastOutputAt / startedAt / createdAt / finishedAt — never updatedAt.
- externalLifecycleRecentRefTime: drop updatedAt from the max().
- staleThreshold gate: for external-lifecycle runs use externalLifecycleRecentRefTime;
  local adapters keep updatedAt (their liveness is pid/group-tracked, not Jobs).

A healthy streaming agent is unaffected (fresh lastOutputAt keeps it out of the
reaper); this only stops updatedAt churn from protecting dead runs, so it cannot
cause false-positive reaping (the 2026-05-06/05-23 RCA concern).

Test (embedded Postgres, TDD — watched it fail first): a pre-adapter
opencode_k8s orphan with ancient startedAt but freshly-churned updatedAt and no
Job is now reaped (process_lost); previously it was skipped indefinitely. Full
heartbeat-process-recovery suite green (85), tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kkroo
kkroo merged commit 486aecc into master Jun 3, 2026
12 checks passed
@kkroo
kkroo deleted the omar/reaper-ignore-updatedat-churn branch June 3, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant