[Gastown] Beads flap in_progress↔open while polecat is heartbeating (container eviction event mishandled)

[jrf0110]

Summary

merge_request and issue beads with active polecat agents oscillate between in_progress and open status repeatedly while the polecat is genuinely heartbeating. This is a systemic regression: at least four beads have failed (5 dispatch attempts each) in the last week as a direct consequence — including, ironically, the bead that was slung to fix this very bug (c09735fe-d2b9-4c81-ae7b-741da708ac0e, which exhausted its dispatch attempts before it could push a single commit).

Symptom

• A polecat is dispatched and starts working on a bead
• The polecat's container heartbeat is fresh (continuous SDK activity)
• The bead's status flips in_progress → open → in_progress → open → ... on a roughly 5-second cadence
• Each cycle counts toward dispatch_attempts; after 5 cycles the bead fails
• Multiple PRs may get opened for the same logical work (each redispatch makes the polecat believe it's starting fresh)

User-observed on towns over multiple days. Confirmed in source.

Root cause

services/gastown/src/dos/town/review-queue.ts:737-754 — the agentCompleted function's "container eviction" branch — unconditionally resets the hooked bead to open and clears assignee_agent_bead_id whenever the runtime fires an agent_completed event with reason: 'container eviction':

} else if (hookedBead && hookedBead.status === 'in_progress') {
  if (input.reason === 'container eviction') {
    // Container eviction: WIP was force-pushed and eviction context
    // was written on the bead body. Reset to open and clear the
    // stale assignee so the reconciler can re-dispatch immediately.
    console.log(
      `[review-queue] agentCompleted: polecat ${agentId} evicted — ` +
        `resetting bead ${agent.current_hook_bead_id} to open`
    );
    updateBeadStatus(sql, agent.current_hook_bead_id, 'open', agentId);
    query(
      sql,
      /* sql */ `
        UPDATE ${beads}
        SET ${beads.columns.assignee_agent_bead_id} = NULL
        WHERE ${beads.bead_id} = ?
      `,
      [agent.current_hook_bead_id]
    );
  }
  // ...
}

On the next 5s alarm tick, reconcileBeads Rule 1 (services/gastown/src/dos/town/reconciler.ts:996-1029) finds the now-open bead, dispatches a fresh polecat → in_progress. The SDK session is durable across container restarts (per cbbd120cf's comments on the analogous reconciler rule), so the same polecat keeps heartbeating the entire time — but every eviction event tears the bead out from under it. With evictions firing on a sub-minute cadence (see "Why now" below), this loops indefinitely.

The fix is identical in spirit to cbbd120cf's defense at services/gastown/src/dos/town/reconciler.ts:910, which the analogous "idle agent hooked to live bead" rule already has:

// reconciler.ts:906-910
// If we've seen a heartbeat in the last 90s, treat the agent as
// alive and leave the hook in place. The 90s window matches
// reconcileAgents' stale-heartbeat threshold, so a truly dead
// agent still gets reaped by the heartbeat path on a later tick.
if (!staleMs(agent.last_activity_at, 90_000)) continue;

The eviction branch in review-queue.ts was never given this defense. So the same eviction event that the reconciler rule treats as transient (do nothing if heartbeat is fresh) is treated as terminal by the agentCompleted path (reset bead, clear assignee).

Why now

Three commits in the last 14 days reshuffled container/auth machinery in ways that probably make eviction events fire more often, even though they didn't touch review-queue.ts itself:

• 90a39936b (May 19) — reshuffled container env hydration
• ba2390f60 (May 19) — removed persisted GASTOWN_CONTAINER_TOKEN
• 3691b1283 (May 20) — bypassed /refresh-token on fresh dispatches

The flap-producing code in review-queue.ts:745 predates all three. The regression is environmental: eviction events fire more often, the latent bug surfaces.

Self-defeating consequence

A bead was slung to fix this exact bug: c09735fe-d2b9-4c81-ae7b-741da708ac0e ("fix(gastown): heartbeat-aware eviction guard in agentCompleted"). It exhausted 5 dispatch attempts and failed without producing any commits. The fix-the-bug bead got eaten by the bug.

This means human intervention is required to land the fix — a polecat cannot reliably complete the work while the bug is in production, because the polecat itself is subject to it.

Recent failed beads attributable to this regression

Bead ID	Title	Date	Outcome
bc85648a-d200-484a-9afa-147f9a904fe2	Babysit PR [0/6]: slingExistingPr Town DO method	2026-05-27	failed at 5 attempts; PR #3536 partially landed
0ee91fff-202e-497f-b484-0182b91d6903	Babysit PR [4/6]: BeadPanel babysit badge	2026-05-27	failed at 5 attempts
c09735fe-d2b9-4c81-ae7b-741da708ac0e	fix(gastown): heartbeat-aware eviction guard	2026-05-27	failed at 5 attempts; the fix bead itself
a41a9206-cee1-4c63-b6b3-ac6b83440bcf	Address review comments on PR #3536	2026-05-27	failed at 5 attempts
b5af666e-57e4-408d-bbe4-582683d02711	Address review comments on PR #3536	2026-05-27	failed at 5 attempts
a3af4c0d-6c74-4d28-9950-ebd0dc670df4	Add 'Gastown' badge to SessionsList	2026-05-22	failed at 5 attempts (matches earlier user report)

Other towns may be affected; the search above only covers one rig.

Diagnostic to confirm

Inside a live gastown container, or via the worker's wrangler tail:

grep '\[review-queue\] agentCompleted: polecat .* evicted —'

One log line per flap. Cross-reference timestamps with the bead's bead_events table — status_changed rows from in_progress to open should align 1:1 with the eviction log lines, with agent_id set to the polecat's id (not 'system').

Fix shape

In services/gastown/src/dos/town/review-queue.ts:737-754, gate the eviction-driven bead reset on staleMs(agent.last_activity_at, 90_000):

} else if (hookedBead && hookedBead.status === 'in_progress') {
  if (input.reason === 'container eviction') {
    if (!staleMs(agent.last_activity_at, 90_000)) {
      // Heartbeat fresh — eviction is transient, the SDK session is
      // recovering. Don't touch the bead. Fall through to unhook the
      // agent (the runtime told us the process exited) but leave the
      // bead in_progress.
      console.log(
        `[review-queue] agentCompleted: polecat ${agentId} eviction event ignored — ` +
          `heartbeat fresh; bead ${agent.current_hook_bead_id} stays in_progress`
      );
    } else {
      // Heartbeat stale — eviction is terminal, reset.
      console.log(
        `[review-queue] agentCompleted: polecat ${agentId} evicted (heartbeat stale) — ` +
          `resetting bead ${agent.current_hook_bead_id} to open`
      );
      updateBeadStatus(sql, agent.current_hook_bead_id, 'open', agentId);
      query(
        sql,
        /* sql */ `
          UPDATE ${beads}
          SET ${beads.columns.assignee_agent_bead_id} = NULL
          WHERE ${beads.bead_id} = ?
        `,
        [agent.current_hook_bead_id]
      );
    }
  }
  // (existing exited-without-gt_done branch unchanged)
}

The unhookBead(sql, agentId) call at line 768 stays unchanged — we still want to unhook the dead agent regardless. The reconciler's Rule 1b (reconciler.ts:1031-1078) handles the "bead has stale assignee but no current_hook" case on the next tick, which is the right cleanup path when the heartbeat truly is stale.

If the agent really is dead, reconcileAgents' stale-heartbeat path (90s window) marks it idle on a later tick, and reconcileBeads Rule 3 resets the bead on its 5-min cadence — by which time we know the eviction was terminal.

The 90s threshold matches the existing reconciler.ts:910 constant. If not already shared, extract HEARTBEAT_STALE_MS = 90_000 to a single location and import from both call sites.

Defense in depth (optional)

services/gastown/src/dos/town/actions.ts (the applyAction for transition_bead, around line 468-479) currently writes the new status without checking whether the SQL row's current status matches the action's from field. If a competing event mutated the status between reconcile-time and apply-time, the apply path silently clobbers the new value. Adding a from-field guard protects against the same class of race in other reconciler rules:

case 'transition_bead': {
  const current = getBead(sql, action.bead_id);
  if (!current) break;
  if (action.from !== undefined && current.status !== action.from) {
    console.warn(
      `[actions] transition_bead skipped: bead ${action.bead_id} status is ` +
        `${current.status}, expected ${action.from} — stale reconciler decision`
    );
    break;
  }
  updateBeadStatus(sql, action.bead_id, action.to, action.actor ?? 'system');
  break;
}

Optional — the primary fix above resolves the user-reported symptom.

Tests

Add to services/gastown/src/dos/town/review-queue.test.ts:

1. Eviction with fresh heartbeat — last_activity_at = Date.now(); call agentCompleted({status: 'completed', reason: 'container eviction'}). Assert: bead is still in_progress, assignee_agent_bead_id unchanged, agent is unhooked + idle.
2. Eviction with stale heartbeat — last_activity_at = Date.now() - 100_000; same call. Assert: bead is open, assignee_agent_bead_id is NULL.
3. Eviction with last_activity_at = null — defensive: treat as stale, reset.

Acceptance criteria

1. review-queue.ts:737-754 consults staleMs(agent.last_activity_at, 90_000) and skips the bead reset when the heartbeat is fresh.
2. The agent is still unhooked + marked idle in either case (the fresh-heartbeat branch only protects the bead, not the agent).
3. The 90s threshold uses the same constant/value as reconciler.ts:910.
4. Three new unit tests cover the three branches.
5. After deploy, the diagnostic log line [review-queue] agentCompleted: polecat ... eviction event ignored should appear in production whenever a heartbeating polecat hits an eviction; flap should stop.

Out of scope

• Investigating why eviction events fire so often (the three recent container/auth commits may be related; that's a separate investigation).
• Hardening the underlying eviction-event source. The platform fires these events; we just need to handle them correctly.
• Backfilling existing flapping/failed beads. The fix is forward-only.
• The optional actions.ts from-field enforcement (defense in depth — separate scope if desired).

Notes for whoever picks this up

The fix likely cannot be made by a polecat — the bug prevents a polecat from completing any non-trivial work while it's in production. Direct human commit + push to gastown-staging is the recommended path. The diff is small (~30 lines including the test additions); the bead c09735fe body is gone but the commit message can simply reference this issue.

References

• Existing defense pattern to mirror: services/gastown/src/dos/town/reconciler.ts:895-925 (commit cbbd120cf)
• Eviction event source: services/gastown/src/dos/Town.do.ts:2055-2090 (agentCompleted RPC)
• Re-dispatch loop: services/gastown/src/dos/town/reconciler.ts:996-1029 (reconcileBeads Rule 1)
• Recent commits that may have made evictions more frequent: 90a39936b, ba2390f60, 3691b1283
• Failed fix bead (self-defeating): c09735fe-d2b9-4c81-ae7b-741da708ac0e