Skip to content

buzz-acp: idle-pool reaper tears down the pool right after a long turn, orphaning background tasks #7171

Description

@Stalker23b

Summary

The idle-pool reaper clocks in buzz-acp (idle-pool sleep and exit-after-inactivity) are refreshed only at dispatch time. A turn that runs longer than the idle bound is torn down on the first reaper tick after it completes: the sidecar kills the CLI process together with its live background tasks and any wake-ups the agent scheduled for after the turn.

Observed incident

Production setup with --idle-pool-sleep 900. An agent ran a 28-minute turn (long build/verification). The turn completed successfully at 17:59:08, and the pool was reaped 13 seconds later on the next reaper tick — last_activity still pointed at the dispatch 28 minutes earlier. Two background tasks the agent was waiting on were orphaned; their completion never woke the agent, which looked like the agent silently going dark for 45+ minutes.

Mechanism

  • last_activity is refreshed when a prompt is dispatched into the pool, but not when a PromptResult comes back or when a steer is acked.
  • Any turn longer than the idle bound therefore guarantees the pool looks idle the moment it finishes.
  • Teardown does not account for live background tasks of the CLI process, and there is no resume/re-wake path after the pool is gone.

Suggested fix

Count turn completion and steer acks as activity when refreshing the idle clocks. Heartbeat results should stay excluded — heartbeat dispatch is gated on an idle slot with no flushable work, so counting them would keep a heartbeat-enabled pool awake forever.

I have a patch running in production (18 agents) since 2026-09-01 with no premature teardowns after long turns; PR incoming.

Known adjacent edges (out of scope for the minimal fix)

  • The in-flight deadline is max_turn_duration + 100s; a turn exceeding it auto-expires and the pool can still be reaped mid-turn.
  • The panic path does not refresh the clocks.
  • Background tasks and scheduled wake-ups still don't keep an otherwise-idle pool alive on their own; that needs a separate awareness/resume mechanism.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions