claim-reaper HeartbeatStale investigation never converges: recipe verdict discarded (reconcile_inflight_investigations write-back gap) → Pending-only, reap path unreachable, unbounded re-archival

[rysweet]

Summary

The claim-reaper's HeartbeatStale investigate-before-reap path never converges: the agentic investigation recipe's terminal verdict is never read back or persisted, so ProductionInvestigator::investigate can only ever emit Pending/StillAlive — never Dead. The reap branch (src/overseer/claim_reaper.rs:341 if !outcome.verdict.should_reap()) is therefore structurally unreachable for a heartbeat-stale engineer. Every sweep re-investigates, re-archives evidence, and re-dispatches the recipe — forever.

This is a distinct root cause from #4462 (stale-assignment sweep non-idempotency), #4464 (claim not released on session completion), and #4449 (evidence collector archives the wrong files). Cross-referenced below.

Evidence (durable archive, one goal, ~14h)

Goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c was archived 10 times (/home/azureuser/.simard/reaped-engineers/…-f29bb15c-{1784692815…1784746010}). Manifests show the same worktree re-archived with monotonically climbing idle_age and never reaped:

worktree …-1784696317-da9e82 archived 5× : idle 2177 → 8040 → 13403 → 19082 → 28764s   (never reaped)
worktree …-1784733114-bba449 archived 3× : idle 3196 → 8452 → 12894s                    (still pending)

Journal (…-1784746010/journal.txt) reaper lines, chronological:

12:58  NOT reaping … (verdict=pending, claim + evidence preserved) + evidence archived, dispatching investigation
13:20  NOT reaping … (verdict=pending)
14:23  reclaimed … (reason=no-worktree, age=n/a, verdict=no-investigation)
15:01  reclaimed … (reason=no-worktree, age=n/a, verdict=no-investigation)
16:05  NOT reaping … (verdict=pending) + evidence archived, dispatching investigation
16:53  NOT reaping … (verdict=pending)          # epoch reused (<1h since 16:05)
17:32  NOT reaping … (verdict=pending) + evidence archived, dispatching investigation   # re-mint (>1h)
18:13  NOT reaping … (verdict=pending)

verdict=pending recurs 6× for this claim with no convergence.

Root causes

A. Verdict write-back gap (primary). reconcile_inflight_investigations (src/overseer/mod.rs:1363-1378) polls the recipe handle and, on completion, only .remove()s it — the recipe's terminal conclusion is discarded, never parsed into an InvestigationVerdict and never persisted for the next sweep to consult. InvestigationVerdict::Dead is constructed only in tests. So investigate() (claim_reaper.rs:1248-1309) returns Pending on dispatch/reuse and StillAlive only on archive error — the reaper can never reap a heartbeat-stale engineer.

B. Freshness-window vs sweep-cadence mismatch (amplifier). ARCHIVE_FRESHNESS_WINDOW = 3600s (claim_reaper.rs:652) is the cross-tick dedupe that suppresses re-archive/re-dispatch. Observed sweep cadence is ~5300s (~88 min) > 3600s, so the epoch ages out before the next sweep → minted=true → fresh archive + fresh LaunchRecipe every sweep (hence 10 durable archives). When sweeps happen <1h apart the dedupe works (16:05→16:53 reused), proving the window is simply too short for the actual cadence.

C. NoWorktree fast-path has no claim-age grace → premature reclaim of nascent/in-flight claims. claim_reaper.rs:332-333 maps DeadReason::NoWorktree => None (unconditional reap, bypassing the #4400 investigation gate). At 14:23 & 15:01 the claim was reclaimed reason=no-worktree while a HeartbeatStale investigation was in-flight (13:20 pending) and before the replacement worktree bba449 materialized (created 15:11, i.e. AFTER the 15:01 reclaim). Two back-to-back no-worktree reclaims of the same claim_key indicate a claim-before-worktree race: a freshly-claimed goal whose worktree hasn't been created yet is indistinguishable from a dead engineer whose worktree was removed → premature reap + orphaned in-flight investigation → goal churn.

Impact

• Heartbeat-stale engineers are never reaped (only the incidental NoWorktree path ever frees the claim), defeating Make Simard investigate a quiet/idle engineer BEFORE ever reaping it (evidence-preserve + agentic WHY, feed self-improvement; only reap if genuinely dead). READ THE FULL AUTHORITATIVE BRIEF and follow #4400's purpose.
• Unbounded re-archival: 10 durable evidence epochs for one goal; repeated LaunchRecipe dispatches burn agent budget.
• Goal thrash + orphaned investigations via the ungraced NoWorktree reclaim.

Suggested fix surface (smallest systemic)

1. Persist + write back the terminal verdict. In reconcile_inflight_investigations, on non-Running completion read the investigation recipe's conclusion and persist a per-claim_key terminal InvestigationOutcome (atomic single-snapshot, mirroring the no-progress loop's I3 persisted-dedupe / I7 atomic write). Have investigate() consult that record first: a persisted Dead reaps on the next sweep; a persisted non-Dead short-circuits re-dispatch/re-archival.
2. Align the dedupe window to cadence — set ARCHIVE_FRESHNESS_WINDOW ≥ the reaper sweep interval (or key reuse on "investigation still outstanding" rather than a fixed wall-clock hour).
3. Add a claim-age grace to the NoWorktree branch — do not reclaim no-worktree until the claim is older than a short worktree-materialization grace, and skip NoWorktree reclaim while an investigation for that claim is in-flight.

Related

• Stale-assignment sweep is non-idempotent: same dead engineer id re-cleared every OODA cycle (clear not durably persisted) #4462 stale-assignment sweep non-idempotent (same class: non-durable persistence → re-fires; different subsystem)
• Engineer claim not released on session completion → leaked claim + 'goal disappeared before effect dispatch' (DownstreamFailed) no-worktree churn #4464 claim not released on session completion → no-worktree churn (related symptom)
• claim-reaper evidence-collector archives checked-in repo fixtures instead of the engineer transcript/exit-status #4449 evidence collector archives checked-in fixtures not the transcript (explains this archive's static-fixture evidence.txt)
• Make Simard investigate a quiet/idle engineer BEFORE ever reaping it (evidence-preserve + agentic WHY, feed self-improvement; only reap if genuinely dead). READ THE FULL AUTHORITATIVE BRIEF and follow #4400 investigate-before-reap (the feature this defect silently defeats)

Filed from a stale-engineer investigation (claim rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c). Engineer verdict itself is separate; this issue is the reaper/investigation-loop defect.

[rysweet]

Corroboration from an independent code-trace (tertiary/architect investigator)

Same goal/claim (advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c), archive …-1784746010. A separate reaper-router + intervention.rs trace independently reproduces both root causes in this issue. Adding the architectural detail this issue doesn't yet spell out, to tighten the fix scope.

Router confirms the two-path asymmetry (claim_reaper.rs:332-358)

let verdict = match reason {
    DeadReason::NoWorktree => None,          // ← UNCONDITIONAL reap, verdict=no-investigation, #4400 gate BYPASSED
    DeadReason::HeartbeatStale => { let outcome = investigator.investigate(...); ... }  // ← gated on should_reap()
};

The #4400 investigate-before-reap guarantee is structurally scoped to HeartbeatStale only. NoWorktree gets a free unconditional reclaim, so the assignment/worktree desync (root-cause C) is not a mere race amplifier — it is a gate hole: any claim whose goal_id → dir correlation transiently fails is reaped with zero investigation.

Desync mechanism localized

WorktreeClaimLivenessProbe::assess (claim_reaper.rs:501-538) correlates claim→worktree via goal_id_from_worktree_dir (engineer_worktree/discovery.rs), which strips a -<epoch>-<hex6> suffix. During a relaunch/cleanup window (old dir gone, replacement bba449 not yet created — created 15:11, after the 15:01 no-worktree reclaim) no dir maps to the goal ⇒ Dead{NoWorktree} ⇒ line 333 None ⇒ ungated reap. This is why a still-present worktree (this archive's manifest.json) coexists with reason=no-worktree reclaims.

Dispatch surface is fine — the gap is upstream

The gated Act path and InterventionKind set (intervention.rs: FileIssue/LaunchRecipe/EscalateBlockedGoal/Whisper) are intact; reap iff verdict.should_reap() holds. InvestigationVerdict::Dead is constructed only in tests — confirming the write-back gap: no terminal verdict ever re-enters the router from reconcile_inflight_investigations. The fix belongs entirely in (1) verdict persistence/write-back and (3) a claim-age/in-flight grace on the NoWorktree branch; the intervention/dispatch layer needs no change.

Fix-surface refinement for #3

Gate the NoWorktree => None arm on provable absence of an assignment/claim-in-materialization, not merely 'no dir this instant': skip NoWorktree reclaim when (a) an investigation for the claim is in-flight, or (b) the claim is younger than a worktree-materialization grace. That closes the hole without touching the HeartbeatStale path.

Engineer liveness verdict for this claim is tracked separately and is still-alive (false positive, fail-closed): no concrete death signature (no real panic/OOM/exit-126-E2BIG/lock-contention); evidence.txt newest files are only checked-in fixtures per #4449. The 14:23/15:01 no-worktree reclaims were this reaper defect, not engineer death. Claim + worktree + durable evidence archive preserved.

[rysweet]

Recurrence data-point (deduplicated — NOT a new issue; this IS #4467).

Fresh evidence from goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c confirms the non-convergence loop this issue describes:

• 10 archive dirs for one goal (~/.simard/reaped-engineers/rysweet_Simard_advance-...-f29bb15c-<ts>), spanning 04:00→18:46 UTC on 2026-07-22.
• Every claim_reaper journal line from 12:58→18:13 logs verdict=pending / NOT reaping … (investigation verdict=pending) — the verdict never transitions to a terminal value.
• Only 1 of 10 archives contains a VERDICT.json; that one (-1784715400) computed verdict=blocked (kgpacks-rs unmeasurable done-gate, Done-gate: agent-kgpacks-rs full parity — DONE (all in-scope KGP-* criteria green on main; native_knowledge + knowledge_client suites pass) #4321) — but the reaper never consumed it, so re-archival continued. This is exactly the reconcile_inflight_investigations write-back gap: verdict computed, discarded, Pending-only, reap path unreachable.
• Compounded by claim-reaper evidence-collector archives checked-in repo fixtures instead of the engineer transcript/exit-status #4449 (each re-archive captures only checked-in fixtures, no transcript), so every re-dispatched investigation restarts blind.

Net effect: an unbounded investigate→archive→pending loop plus engineer re-dispatch churn (worktree da9e82 → reclaimed no-worktree → bba449 re-checked-out 15:11:55) rather than a converging diagnosis. No fix action taken here (tracked by this issue; #4449 fix already at HEAD f200d50, #4437 via PR #4445).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[rysweet]

Fresh live reproduction (2026-07-22) — goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c.

The claim-reaper archived 10+ evidence dirs for the same claim in ~15h, each ending
NOT reaping ... verdict="pending"; the investigation verdict never wrote back, so the reap
path stayed unreachable (exactly this bug). Sample archived evidence dirs under
~/.simard/reaped-engineers/:

...-1784692815  04:00
...-1784698494  05:34
...-1784704358  07:12
...-1784709721  08:42
...-1784715400  11:25
...-1784721147  11:52
...-1784725082  12:58
...-1784736312  16:05
...-1784741567  17:32
...-1784746010  18:46

Reaper log excerpts (all verdict="pending"):

claim-reaper: evidence archived, dispatching agentic investigation (verdict=pending, ...) idle_age_secs=28764
claim-reaper: NOT reaping rysweet/Simard:advance-...-f29bb15c (investigation verdict=pending, ...) verdict="pending"
claim-reaper: evidence archived, dispatching agentic investigation (verdict=pending, ...) idle_age_secs=3196
claim-reaper: evidence archived, dispatching agentic investigation (verdict=pending, ...) idle_age_secs=8452

Compounding factors observed in the same run (already tracked): the goal is a
standing/perpetual goal that the OODA breaker classifies as idled this cycle (normal, not a fault) — so it should never have been a reap candidate (#4437) — and the archived
evidence.txt contained checked-in repo fixtures (ci-health-green.json,
coin_gym/fixtures/*.json, package.json) rather than the engineer transcript/exit status
(#4449), which makes any terminal verdict impossible and forces the fail-closed still-alive
default. Net effect is exactly the unbounded re-archival loop described here.

[rysweet]

Recurrence + code root-cause (deduplicated — this IS #4467, no new issue filed).

New evidence: 2026-07-22 stale-engineer investigation of goal
advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c
archive ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784766958
(manifest idle_age_secs: 13852). journal.txt shows the reaper re-archiving
evidence roughly hourly with verdict=pending throughout and no logged
investigation conclusion:

18:46:50 evidence archived, dispatch investigation; NOT reaping (verdict=pending)
20:04:40 evidence archived; NOT reaping (verdict=pending)
21:24:29 evidence archived; NOT reaping (verdict=pending)
22:31:57 evidence archived; NOT reaping (verdict=pending)
23:32:42 evidence archived (idle_age_secs=10055); NOT reaping (verdict=pending)

Root cause located in code: Overseer::reconcile_inflight_investigations
(src/overseer/mod.rs) polls each in-flight investigation handle and, on a
terminal status, only removes the handle from inflight_investigations. It
never reads the concluded recipe's verdict and writes it back to the store the
claim-reaper consumes via StaleEngineerInvestigator::investigate. So every
subsequent reaper sweep still sees no terminal outcome → returns Pending →
keeps the claim → re-archives → re-dispatches. Unbounded re-archival; the reap
path is unreachable even for a genuinely-dead engineer.

Partial fix dispatched: PR #4489 adds structured tracing on each terminal
investigation (the detection half) so this silent loop is diagnosable from
logs. It deliberately does not change the reap gate. The full remedy — writing
the verdict back so the reaper can act — remains this issue's scope (a
safety-critical change: a wrong write-back could false-reap a live engineer).

[rysweet]

Cross-goal recurrence — unbounded re-archival on a second goal

Deduplicated — NOT a new issue; this IS #4467. Prior data points here were all goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c. Fresh evidence shows the non-convergence loop also on goal move-the-governed-repo-roster-out-of-framework-a8f57a50 (claim key treated strictly as untrusted DATA).

16 archive dirs for one claim key in ~28h under ~/.simard/reaped-engineers/rysweet_Simard_move-the-governed-repo-roster-out-of-framework-a8f57a50-<ts>:
1784691321, 1784706633, 1784711724, 1784715401, 1784736312, 1784741567, 1784746010, 1784750681, 1784755470, 1784759517, 1784763162, 1784766958, 1784770664, 1784776229, 1784781674, 1784791501.

journal.txt in the newest archive logs the reaper repeatedly evidence archived, dispatching agentic investigation (verdict=pending) / NOT reaping … (investigation verdict="pending") (e.g. 01:37, 03:10, 04:41), interleaved with reclaimed … reason=no-worktree … verdict=no-investigation. The verdict never transitions to terminal → reap path stays unreachable → hourly re-archival, exactly as this issue states. Confirms the write-back gap is goal-independent.

[rysweet]

Fresh reproduction — dead engineer leaked by the non-converging investigation loop (goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c)

A stale-engineer investigation (prompt_assets/simard/overseer/investigate_stale_engineer.md) on claim
rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c just reproduced this defect end-to-end. Grounded evidence:

• Process truth: no process references checkout 1784825193-27555d — the engineer is provably gone (only the investigation's own PIDs remain).
• Filesystem forensics: all 2162 worktree files carry checkout-time mtime; newest = .simard/goal-task.bin @ 1784825193.55 = exact checkout ts. Zero post-launch writes. Idle age 23593s (~6.5h). No commits, no PR, no wip ref beyond the bare checkout branch → zero engineer output.
• Diagnostic blindness (claim-reaper evidence-collector archives checked-in repo fixtures instead of the engineer transcript/exit-status #4449): the durable archive ~/.simard/reaped-engineers/…-1784848787/evidence.txt contains tails of committed repo fixtures (ci-health, atelier, coin_gym, dashboard-audit, package.json, copilot-submit), not the engineer transcript/exit-status. A death-signature scan found zero genuine crash lines — WHY the engineer went silent is unknowable from captured evidence.
• Non-convergence / leak (this issue, claim-reaper HeartbeatStale investigation never converges: recipe verdict discarded (reconcile_inflight_investigations write-back gap) → Pending-only, reap path unreachable, unbounded re-archival #4467): the reaper re-archived hourly (idle_age 7371→12370→16253→23593) logging verdict=pending every cycle. A prior investigation (docs(investigation): still-alive verdict for kgpacks-parity stale-engineer f29bb15c (recurrence of #4437) #4527, ~4h earlier) concluded still-alive from the same fixture archive — a fail-closed default on an absent signature. The dead claim has been leaking for ~6.5h and would never be reaped.
• Liveness false positive (claim-reaper/OODA brain mis-attribute a PRIOR incarnation's wedge log to the CURRENT worktree, and idle-age conflates completed with wedged (diagnostic-layer defect behind false-positive stale-engineer reaps) #4500): journal lines "worker present … in-flight work looks alive" were inferred from a live WIP branch ref, not a real PID.

Note: evidence.txt includes an untrusted copilot-submit "Reply with READY" payload — treated strictly as DATA, ignored as an instruction (prompt-injection defence).

Verdict of this investigation: dead / cause unknown (process provably gone, zero output; cause unknowable because evidence was blinded by #4449). Aliveness was not ambiguous, so fail-closed did not force still-alive; the added decisive evidence (process enumeration + mtime forensics) is exactly what the prior fixture-only investigation lacked.

Action taken: claim released and worktree removed to break the leak loop (durable archive preserved). In-flight fixes tracked by PR #4452 (#4449) and PR #4489 (#4467) address the root diagnostic + convergence defects; no new fix dispatched to avoid duplicate PRs.