fix(buzz-acp): distinguish idle vs hard-cap timeout, dead-letter hard kills immediately#1844
Merged
Merged
Conversation
… kills immediately
Hard-cap timeouts (wall-clock limit) were mislabeled as idle-timeout inactivity
and then retried up to 10× — each cycle re-running the same >1h task from a fresh
session before dying again, producing ~10h of silent churn.
Three coupled fixes:
- Plumb `TimeoutKind { Idle, Hard }` through `PromptOutcome::Timeout` so every
downstream site can tell which clock fired. Split the combined cancel-drain
`IdleTimeout | HardTimeout` arm to carry the correct kind through.
- Dead-letter hard-cap kills immediately in `handle_prompt_result` without calling
`queue.requeue()`. Hard-cap failure is deterministic; a fresh session will
reproduce the same death. Idle timeout keeps its current retry behavior (may be
a transient provider stall). Observer `outcome_label` now emits `"idle_timeout"`
or `"hard_timeout"` instead of `"timeout"` for observability.
- Raise `max_turn_duration` default from 3600s to 7200s so genuinely long turns
(heavy implementation work) have headroom. Idle stays at 900s, preserving the
`idle < max_turn` invariant and the real hang detector.
Adds unit tests verifying hard timeout is NOT requeued and idle IS, and that
`outcome_label` differs between the two kinds.
The queue's IN_FLIGHT_DEADLINE_SECS (3700) was hardcoded to the old 3600s max_turn_duration + 100s buffer. With the cap raised to 7200s, a legitimate 2h turn hits the 3700s backstop at ~62 min and gets force-released while still running — enabling duplicate same-channel processing. Replace the static constant with a configurable `in_flight_deadline` field on EventQueue, derived from max_turn_duration_secs + 100s buffer at the prod construction site. Tests keep the default (7300s) via EventQueue::new(); prod calls .with_in_flight_deadline(). Also: harden the hard-timeout requeue test with a real event batch instead of an empty FlushBatch, and add invariant tests asserting in_flight_deadline > max_turn_duration.
Reject max_turn_duration above 604800s (7 days) at the config boundary so the unchecked u64 addition in with_in_flight_deadline cannot overflow. Make the requeue test assert queued event counts (not just channel keys) to verify event preservation on idle timeout and event drop on hard timeout.
…-letter log The hard-timeout match arm in handle_prompt_result used unreachable!(), which would panic the main loop if the else-if chain were ever reordered. Replace with a graceful fallback returning the honest reason string. Add a tracing::error! log line in the hard-cap dead-letter arm to match the retries-exhausted dead-letter path — ops greps for dead-letters now catch both paths symmetrically.
brow
pushed a commit
that referenced
this pull request
Jul 14, 2026
…image-paste * origin/main: refactor(clients): standardize product naming on community (#1858) fix(desktop): retain live rows after window exhaustion (#1810) Add private product feedback sidecar (#1857) fix(desktop): resolve Doctor install shell and command detection on Windows (#1854) fix(desktop): navigate to channel from inbox thread header (#1847) feat(desktop): surface needsRestart badge on live UI surfaces (#1853) fix(desktop): prevent menu-to-dialog UI lockups (#1839) feat(relay): add durable community archival (#1834) feat(desktop): add conversation-style DM composer (#1768) feat(push): add public APNs gateway (#1770) feat(relay): add atomic community ownership transfer (#1845) fix(desktop): stabilize agent identity restore (#1831) fix(buzz-acp): distinguish idle vs hard-cap timeout, dead-letter hard kills immediately (#1844) fix(observer): align scroll-anchor ids with transcript display-block keys (#1849) [codex] Add view activity label to agent popover (#1748) ci(desktop): surface flaky E2E tests instead of retry-masking them (#1838) fix(desktop): treat channel creator as member before 39002 provisioning (#1830) fix(mobile): mirror app bar title padding when actions are empty (#1832) Co-authored-by: npub1shglkdhngx3hrnhf4gf8vhpqdrmeludctechdvpwd3988zzs7ncq2cmtxu <85d1fb36f341a371cee9aa12765c2068f79ff1b85e7176b02e6c4a738850f4f0@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub1shglkdhngx3hrnhf4gf8vhpqdrmeludctechdvpwd3988zzs7ncq2cmtxu <85d1fb36f341a371cee9aa12765c2068f79ff1b85e7176b02e6c4a738850f4f0@sprout-oss.stage.blox.sqprod.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hard-cap timeouts (wall-clock limit) were mislabeled as idle-timeout "inactivity" and retried up to 10× — each cycle re-running the same >1h task from a fresh session before dying again, burning ~10h of compute on a single message. Additionally, the queue's in-flight backstop was hardcoded to the old 3600s cap + 100s buffer, meaning a legitimate 2h turn would get force-released at ~62 min while still running.
Five coupled fixes:
Plumb
TimeoutKind { Idle, Hard }throughPromptOutcome::Timeoutso every downstream site can distinguish which clock fired. Split the combinedIdleTimeout | HardTimeoutcancel-drain arm inpool.rsto carry the correct kind. Observeroutcome_labelnow emits"idle_timeout"or"hard_timeout"instead of the ambiguous"timeout".Dead-letter hard-cap kills immediately in
handle_prompt_resultwithout callingqueue.requeue(). Hard-cap failure is deterministic — a fresh session reproduces the same death. Idle timeout keeps its current retry behavior (may be a transient provider stall). The user-facing message now reads"Agent turn exceeded the maximum duration ({N}s)"instead of the misleading"Agent session timed out due to inactivity".Raise
max_turn_durationdefault 3600s → 7200s so genuinely long turns (heavy implementation work) have headroom. Idle stays at 900s, preserving theidle < max_turninvariant and the real hang detector.Derive in-flight deadline from
max_turn_durationinstead of a hardcodedIN_FLIGHT_DEADLINE_SECS = 3700. The queue'sEventQueuenow carries a configurablein_flight_deadlinefield set tomax_turn_duration + 100sat the prod construction site. Tests keep the default (7300s) viaEventQueue::new(); prod calls.with_in_flight_deadline(). AddsDEFAULT_MAX_TURN_DURATION_SECSconstant toconfig.rsand wires it into all test fixtures and the clapdefault_value_t.Validate
max_turn_durationupper bound at the config boundary (MAX_TURN_DURATION_CEILING_SECS = 604_800, 7 days). Rejects values above the ceiling with aConfigError, preventing arithmetic overflow in the in-flight deadline derivation (max_turn + 100). The timeout requeue test (hard_timeout_not_requeued_idle_timeout_is_requeued) now asserts queued event counts viaqueued_event_count(), not just channel-key presence, verifying that idle timeout preserves events and hard timeout drops them.Files changed
crates/buzz-acp/src/pool.rs—TimeoutKindenum, split cancel-drain arm,outcome_labelbranchingcrates/buzz-acp/src/lib.rs—handle_prompt_resulthard-cap divert,death_message/outcome_label/ dead-letterreasonsplit, prodEventQueueconstruction with.with_in_flight_deadline(), hardened timeout requeue test with real event batch and event-count assertionscrates/buzz-acp/src/config.rs—DEFAULT_MAX_TURN_DURATION_SECSandMAX_TURN_DURATION_CEILING_SECSconstants, upper-bound validation withConfigError,default_value_ton clap arg, boundary testscrates/buzz-acp/src/queue.rs—in_flight_deadlinefield onEventQueue,with_in_flight_deadline()builder,queued_event_count()test accessor, replaced allIN_FLIGHT_DEADLINE_SECSusage with instance field, invariant tests