feat(automation): retry transient backend errors, surface autonomy#338
Conversation
From an 11-day forensic audit of the live automation loop (534 successful runs, 19 failures — all transient codex app-server errors): - New run_agent_task_with_retry at the AgentTaskBackend boundary: up to two bounded retries (2s/5s backoff) for retryable classifications only (Timeout, Unavailable — the observed "timed out waiting for codex app-server" and "closed stdout before completing" shapes), never exceeding the job's timeout_secs budget; malformed output remains non-retryable by contract. Routed through all four call sites (memory curator, combined review, user jobs, lifecycle fallback). - Autonomy surfaced per user directive (automation needs no human approval): require_dashboard_approval was already deprecated in code (parsed, never read); config get now prints effective_apply_policy: autonomous with per-category apply behavior, the patch field carries a deprecation doc, and the loop-contract docs state autonomous apply explicitly. - Retry semantics pinned by four zero-backoff FlakyBackend tests; the five fallback/failed-ledger integration tests short-circuit retries via timeout_secs: 1 so the suite stays fast. Verified: automation_runner_test 143/143 (twice), dashboard automation config tests green, clippy and fmt clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afd0c7605b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| let backoff = policy.backoff_before_attempt(attempt + 1); | ||
| // Respect the overall job timeout: never sleep/retry past budget. | ||
| if start.elapsed().saturating_add(backoff) >= policy.budget { |
There was a problem hiding this comment.
When a retryable failure happens quickly, such as the app-server closing stdout, this check only budgets the sleep before the next attempt, then starts another CodexAppServerBackend call whose own timeout is still the full timeout_secs (CodexAppServerBackend::new sets config.timeout from that value). With the default 60s budget, two quick failures followed by a hung third attempt can run for about 67s, so the automation job no longer honors its configured timeout; actual Timeout errors also consume the whole first-attempt deadline and therefore never get retried. Please reserve time for the next attempt or wrap the whole retry loop in the overall budget.
Useful? React with 👍 / 👎.
1 participant
Bounded transient-only retry at the single backend boundary (evidence: 19/19 observed automation failures over 11 days were transient app-server errors), reusing existing retryable classifications and honoring the job timeout budget. Autonomy made explicit per user directive: effective_apply_policy surfaced in config get, the already-deprecated require_dashboard_approval documented as such, docs updated.
🤖 Generated with Claude Code