fix(tui): retry sub-agent API timeouts with backoff; raise default timeout to 600s

[Hmbown]

Problem

Dogfood-confirmed 2026-08-03: a 6-agent fan-out was wiped out one by one — every sub-agent died with Sub-agent interrupted before completion (API call timed out after 120000ms; checkpoint preserved for continuation).

Root cause: in crates/tui/src/tools/subagent/mod.rs, the per-step create_message call is wrapped in tokio::time::timeout(step_api_timeout, …), and the timeout arm returned Interrupted with zero retries. Transient provider errors already retry (SUBAGENT_TRANSIENT_PROVIDER_MAX_RETRIES), but timeouts did not — so one live-but-slow provider call killed an entire child. Full findings: codewhale-ops/FINISH-0.9.4.md entries #39/#40.

kimi-code comparison

A scout compared against kimi-code (packages/agent-core): it retries every provider step up to 10 attempts with exponential backoff 0.5s→32s + 25% jitter, honors Retry-After, classifies timeouts as retryable, uses streaming so live-but-slow calls never hit a wall-clock cap, and has first-class resume. Owner directive: "learn from kimicode."

What this PR does

(a) Timeout arm is now retryable. A per-step API timeout folds into the retry machinery:

• New budget SUBAGENT_API_TIMEOUT_MAX_RETRIES = 5 per step.
• Exponential backoff: base 1s, ×2 per retry, capped at 30s, ±20% jitter (UUID-entropy idiom shared with llm_client::RetryConfig) so a fan-out that times out together doesn't re-fire in lockstep.
• Emits the same style of ModelWait progress event as the existing "retrying API request" transient path, so the TUI shows the retry (…: API call timed out after Nms; retrying API request k/5 in Xms).
• After exhaustion, behavior is unchanged: Interrupted with checkpoint_reason: "api_timeout" and the checkpoint preserved for continuation.

(c) Config limits (crates/tui/src/config/subagent_limits.rs):

• DEFAULT_SUBAGENT_API_TIMEOUT_SECS: 120 → 600 (matches the mitigation operators already applied by hand).
• Clamp ceiling MAX_SUBAGENT_API_TIMEOUT_SECS: 1800 → 3600.
• Doc comments updated, including the stale ~/.deepseek/config.toml reference (product home is ~/.codewhale); docs/SUBAGENTS.md, docs/CONFIGURATION.md, and config.example.toml updated to match.
• Knock-on effect (test updated, no code change): with the 600s API default, the heartbeat resolver's "at least api_timeout + 30s" floor lifts the resolved default heartbeat from 300s to 630s (subagent_heartbeat_timeout_defaults_clamps_and_respects_api_timeout now asserts the resolved value; docs updated).

Tests

• Backoff sequence is deterministic and bounded: 1s/2s/4s/8s/16s then capped at 30s; jittered delay stays within ±20% of the base (32 samples per retry level).
• Integration: server slow on attempt 1 only → retried exactly once, then Completed (subagent_retries_api_timeout_before_succeeding).
• Integration: server slow on every attempt → exactly 1 + SUBAGENT_API_TIMEOUT_MAX_RETRIES calls, then Interrupted with api_timeout checkpoint, waiting_for_user projection, no parking (updated api_timeout_preserves_checkpoint_and_returns_needs_input_without_parking; new always_delayed_chat_client helper mirrors the existing fake-server pattern; production backoff is shrunk via a #[cfg(test)] runtime knob so the test runs in milliseconds).
• Config: default is 600, 3600 accepted, 3601 clamps to 3600 (added literal assertions to subagent_api_timeout_defaults_and_clamps).
• child_runtime()/background_runtime() preserve the new backoff field (extended child_and_background_runtimes_preserve_step_api_timeout).
• Gates: cargo fmt --all clean; cargo test -p codewhale-tui — 9621 passed; the only 19 failures are pre-existing environment-dependent tests (missing DeepSeek/MiniMax/Anthropic keys, provider-catalog fixtures) verified to fail identically on base d53f4f998.

Not covered at unit level: the exact wall-clock interleaving of timeout → progress event → sleep inside request_subagent_model_response_with_retries (exercised end-to-end by the two integration tests instead); the TUI rendering of the progress event (same event path as the existing transient retry).

Follow-ups (NOT in this PR)

• (b) Idle-based streaming keepalive for sub-agent calls, reusing the engine's stream_chunk_timeout (crates/tui/src/core/engine.rs:467), so a live-but-slow streaming call never trips a wall-clock cap at all (kimi's actual defense-in-depth).
• (d) Automated checkpoint resume on re-dispatch (kimi's resume=<agent_id> semantics) instead of relying on the parent to re-dispatch manually.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[Copilot]

Pull request overview

This PR improves sub-agent reliability in the TUI by treating per-step create_message API timeouts as retryable failures (with exponential backoff + jitter) and by increasing the default/configured timeout ceilings to better accommodate slow-but-live reasoning calls.

Changes:

• Add timeout-retry behavior for sub-agent per-step API calls, including exponential backoff + jitter and progress events during retries.
• Raise the default sub-agent API timeout to 600s and increase the configured maximum to 3600s, updating related config resolution behavior and tests.
• Update documentation to reflect the new defaults/retry behavior and the knock-on heartbeat resolution change.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
docs/SUBAGENTS.md	Updates sub-agent timeout guidance and explains the new default/behavior.
docs/CONFIGURATION.md	Updates configuration docs for sub-agent API/heartbeat timeouts and retry behavior.
crates/tui/src/tools/subagent/tests.rs	Adds/updates integration + unit tests for timeout retries/backoff and runtime propagation.
crates/tui/src/tools/subagent/mod.rs	Implements timeout retry budget + backoff/jitter, adds runtime field for test backoff shrinking.
crates/tui/src/core/engine.rs	Updates sub-agent API timeout clamp documentation in engine config.
crates/tui/src/config/tests.rs	Updates config tests for new defaults and clamping behavior.
crates/tui/src/config/subagent_limits.rs	Raises API timeout defaults/max constants used for clamping.
crates/tui/src/config.rs	Updates config docs and resolution behavior to match new sub-agent timeout defaults.
crates/tui/src/commands/groups/config/config.rs	Updates config command test expectations for resolved sub-agent timeout values.
config.example.toml	Updates example config to show the new default/clamp values.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.