Deterministic 400 invalid_request_error from an orphaned tool_use_id is classified retryable and never repaired: session re-sends the identical rejected request for 6+ hours

[etafund]

Summary

A single orphaned tool_use_id in a session's message history makes every subsequent Anthropic request fail with a deterministic HTTP 400 invalid_request_error, and prime-agent classifies that 400 as retryable on the first failure and never repairs the offending tool_use/tool_result pairing. The result observed in the field: the identical rejected payload was re-sent 28 times across 14 turn attempts over 6 h 18 m, and the session only stopped burning requests when its worker was killed externally.

Environment

prime-agent	0.7.1 at time of incident; 0.7.2 currently installed
Node.js	v22.22.1
OS	Linux
Provider / API	Anthropic, anthropic-messages
Models affected	claude-fable-5, claude-opus-5 (two independent sessions, same day)
Retry settings	defaults (no retry override in ~/.prime/agent/settings.json)

The retry classifier is textually identical in the 0.7.1 and 0.7.2 shipped bundles (_isRetryableError, _isStructuredPermanentProviderFailure and _isStructuredPermanentProviderRetryExhausted all match), so this is not fixed in 0.7.2.

Observed behavior

Exact API error

Verbatim provider response body, as captured in the cause field of the ai.provider / provider stream failure record in ~/.prime/agent/logs/agent.jsonl:

400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.464.content.2: unexpected `tool_use_id` found in `tool_result` blocks: toolu_01JEKX4JYYoWzqYmRQ4oJLjd. Each `tool_result` block must have a corresponding `tool_use` block in the previous message."},"request_id":"req_<redacted>"}

Verbatim prime-agent log record (one of 28; requestId redacted, pid/mode dropped):

{"provider":"anthropic","model":"claude-fable-5","api":"anthropic-messages","kind":"invalid_request","providerErrorType":"invalid_request_error","status":400,"requestId":"req_<redacted>","message":"Provider rejected the request (invalid_request_error, 400): messages.464.content.2: unexpected `tool_use_id` found in `tool_result` blocks: toolu_01JEKX4JYYoWzqYmRQ4oJLjd. Each `tool_result` block must have a corresponding `tool_use` block in the previous message. [request_id: req_<redacted>]","ts":"2026-08-07T19:17:24.566Z","level":"error","component":"ai.provider","msg":"provider stream failure"}

Every one of the 28 records carries a distinct request_id, meaning these are 28 real HTTP round trips, not 28 log lines for one request. All 28 name the same offset messages.464.content.2 and the same toolu_01JEKX4JYYoWzqYmRQ4oJLjd — the payload never changed between attempts.

Timeline

Each turn attempt = 1 initial request + 1 auto-retry ~2.4 s later, then the turn ends in error. Nothing repairs state, so the next wakeup of the session repeats the pair.

#	first request (UTC)	auto-retry (UTC)	retry delay	gap since previous attempt
1	2026-08-07T19:17:24.566Z	19:17:26.961Z	2.4 s	—
2	2026-08-07T19:18:56.133Z	19:18:58.843Z	2.7 s	92 s
3	2026-08-07T19:20:01.263Z	19:20:03.626Z	2.4 s	65 s
4	2026-08-07T19:20:52.800Z	19:20:55.200Z	2.4 s	52 s
5	2026-08-07T19:26:38.252Z	19:26:40.647Z	2.4 s	345 s
6	2026-08-07T19:29:51.104Z	19:29:53.499Z	2.4 s	193 s
7	2026-08-07T19:31:00.136Z	19:31:02.460Z	2.3 s	69 s
8	2026-08-07T19:33:52.842Z	19:33:55.429Z	2.6 s	173 s
9	2026-08-07T19:35:55.389Z	19:35:57.889Z	2.5 s	123 s
10	2026-08-07T19:55:26.280Z	19:55:28.855Z	2.6 s	1171 s
11	2026-08-07T21:05:55.598Z	21:05:57.912Z	2.3 s	4229 s
12	2026-08-07T22:35:55.313Z	22:35:57.627Z	2.3 s	5400 s
13	2026-08-08T00:05:55.349Z	00:05:57.738Z	2.4 s	5400 s
14	2026-08-08T01:35:55.533Z	01:35:57.879Z	2.3 s	5400 s

• 28 rejected requests / 14 turn attempts, span 2026-08-07T19:17:24.566Z → 2026-08-08T01:35:57.879Z = 6 h 18 m 33 s.
• Attempts 11–14 are spaced 5400 s (90 min) to the second — a periodic session wakeup, not user input, kept re-driving the poisoned state unattended.
• prime-agent never repaired the message list, never escalated to a terminal session error, and never stopped re-sending. The loop ended only because the session worker received SIGTERM at 2026-08-08T02:08:02Z.
• A second, independent session the same day hit the same defect on claude-opus-5 with a different orphan (toolu_01RR3tGsdrhhMWAqtLLYMPWH at messages.4.content.2), 4 rejected requests across 2 turn attempts — so this is not specific to one long session or one model lane.

Expected behavior

1. A 400 invalid_request_error is deterministic and non-retryable. Re-sending an identical body must not happen; the retry should be suppressed entirely, not merely capped on the second failure.
2. A tool_use_id-pairing 400 identifies a self-inflicted, locally repairable defect in the outgoing message list. prime-agent should repair the conversation state (drop the tool_result block(s) with no surviving tool_use, or synthesize the missing tool_use) and retry once with the corrected payload.
3. If repair is not possible, the session must fail terminally and visibly, and must not be re-driven by later wakeups (heartbeat / idle sweep / resume) into re-sending a payload already known to be rejected.

Root cause hypothesis

Two independent defects compound.

(a) The retry classifier does not consider HTTP status class. _isRetryableError(message) returns true for any assistant message with stopReason === "error" and a non-empty errorMessage, except for four carve-outs: context overflow, the faux provider's queue exhaustion, an agent_lifecycle_failure diagnostic, and _isStructuredPermanentProviderRetryExhausted(message). There is no check on status, no 4xx-vs-5xx distinction.

_isStructuredPermanentProviderFailure does already recognise kind === "invalid_request" (alongside "auth" and "refusal") as permanent — the classification exists and is correct. But the only consumer inside _isRetryableError is _isStructuredPermanentProviderRetryExhausted, which gates it behind this._retryAttempt > 0. So the first occurrence of a known-permanent failure is unconditionally treated as transient. Net effect: exactly one wasted retry of a deterministic 400 per turn — visible as the 2.4 s pairs above.

(b) There is no conversation-state repair path. _handleRetryableError prepares for the retry by dropping the trailing assistant message:

const messages = this.agent.state.messages;
if (messages.length > 0 && messages[messages.length - 1].role === "assistant") {
  this.agent.state.messages = messages.slice(0, -1);
}

It never inspects tool_use/tool_result pairing, so the rebuilt request is byte-identical in the part the API rejected. Nothing else in the turn lifecycle repairs it either, and _retryAttempt resets between turns, so the permanent-failure knowledge from (a) is discarded — every later wakeup starts the same 2-request cycle from scratch. The session is wedged for as long as it exists.

Likely producer of the orphan, and the feedback loop. #984 (closed not_planned) describes transformMessages dropping assistant messages with stopReason error/aborted while leaving their toolResult messages in place, which orphans exactly this way. That closes a loop: an errored turn orphans its tool results → every subsequent request is a deterministic 400 → those turns also end in stopReason: "error" → the orphan is never cleaned up. A single transient failure mid-tool-call permanently poisons the session. Fixing #984 removes one producer; the retry/repair defects here are what turn a one-off orphan into an unbounded, unattended request loop, so they are worth fixing independently.

Reproduction sketch

Deterministic, no flaky timing needed.

1. Build a session whose next outgoing request contains a tool_result block whose tool_use_id does not match any tool_use block in the preceding assistant message. Either:
   • directly — hand-edit the persisted session message list to change one toolResult.toolCallId to a nonexistent id (e.g. toolu_00000000000000000000000000); or
   • through the real path — start a tool call, abort or error the assistant turn after the tool call is emitted and its result recorded, then continue the session (the transformMessages orphans tool results when the parent assistant turn was aborted or errored #984 path).
2. Send any prompt on that session against the Anthropic provider.
3. Observe in ~/.prime/agent/logs/agent.jsonl: two component: "ai.provider", msg: "provider stream failure" records with status: 400, kind: "invalid_request", distinct requestIds, ~2.4 s apart, both naming the same tool_use_id.
4. Send another prompt (or let a heartbeat fire). Observe the identical pair again. Repeat indefinitely — the session never recovers.

Suggested fix

1. In _isRetryableError, return false for _isStructuredPermanentProviderFailure(message) regardless of _retryAttempt — that is, use _isStructuredPermanentProviderFailure directly, not the _retryAttempt > 0-gated _isStructuredPermanentProviderRetryExhausted. Keep a status-class guard as a backstop: any provider stream failure with status in 400..499 other than 408/429 — and other than 401/403, which are deliberately retried once via the existing auth-refresh path — is non-retryable.
2. Add a repair pass before a request is sent (or on a tool_use_id-mismatch 400): scan the outgoing message list and drop every tool_result block whose tool_use_id has no matching tool_use in the preceding assistant message; symmetrically, synthesize a tool_result for any tool_use with no result. Repair once, resend once, and only then fail.
3. When repair is impossible, surface a terminal session error rather than an ordinary turn error, and mark the session so later wakeups (heartbeat, idle sweep, resume) do not re-issue the same rejected payload.

Acceptance criteria

• A provider stream failure with kind: "invalid_request" / status: 400 produces exactly one HTTP request. _isRetryableError returns false on the first such failure; no auto_retry_start is emitted for it.
• A 400 whose message matches the tool_use_id / tool_result pairing error triggers a state-repair pass; the retried request contains no tool_result block without a matching tool_use, and the session makes forward progress.
• Given the reproduction above, total requests to the provider for a wedged session over 10 heartbeat cycles is bounded (repair succeeds, or the session is terminally failed) — never 2 × cycles as observed here.
• Unit test: _isRetryableError returns false for a first-occurrence message carrying a provider_stream_failure diagnostic with kind: "invalid_request", and for status 400/404/422; still true for 408, 429, 500, 502, 503, 529; the existing 401/403 auth-refresh retry path is unchanged.
• Unit test: a message list containing a toolResult with an unmatched toolCallId is normalized to a list the Anthropic Messages API accepts.

Related: #984 (transformMessages orphans tool results when the parent assistant turn was aborted or errored) — closed not_planned; that path is a plausible producer of the malformed state reported here.

Addendum (further investigation)

Follow-up reading of the current main sources refines the three points below. All references are to functions in repo-relative TypeScript sources.

1. The classification is already correct; the gate in front of it is not. classifyStreamFailure in packages/ai/src/utils/stream-failure.ts maps HTTP 400 to the kind "invalid_request", so a rejected request reaches the session already labelled structurally permanent. _isStructuredPermanentProviderFailure in packages/coding-agent/src/core/agent-session.ts honours that label for "auth", "invalid_request" and "refusal". The defect is the gate: _isRetryableError consults that predicate only indirectly, through _isStructuredPermanentProviderRetryExhausted, which additionally requires _retryAttempt > 0. On the first failure of a dispatch the retry counter is zero, so the permanence signal is discarded and a structurally-permanent failure is always retried exactly once per dispatch. That is bounded within a single turn, but it is not bounded across the lifetime of a wedged session: every scheduled re-dispatch starts from _retryAttempt === 0 again, so the same one-wasted-retry pattern repeats, and the total compounds into dozens of wasted, guaranteed-to-fail API calls for a payload whose rejection is already known to be deterministic.

2. _isRetryableError is default-allow. Its final statement returns true; permanence has to be proven by one of a small set of carve-outs (context overflow, the faux provider's queue exhaustion, an agent_lifecycle_failure diagnostic, and the retry-exhausted check above). Nothing in that set covers the kind "unknown", which classifyStreamFailure assigns whenever a provider error type and status cannot be matched to a known class, so every "unknown" failure falls through to retryable unconditionally. That is not a rare residual bucket: in one deployment's failure records, "unknown" was by far the largest single class of retried errors. Inverting the default (retry only what is positively known to be transient) would also cover the case above as a side effect.

3. The resend path still performs no pairing repair. _handleRetryableError in packages/coding-agent/src/core/agent-session.ts prepares the retry by dropping the trailing assistant message and nothing else. It never inspects tool_use / tool_result pairing before the re-send, so the part of the payload the API actually rejected is reconstructed unchanged and a single malformed tool_use_id wedges the session permanently. This is the same shape addressed by PR #991 (fix(ai): drop tool results whose parent tool call was aborted), which was closed without merging. It also complements issue #1384 ("Bound transcript repair, compaction, and autonomous recovery"): that issue scopes the state-repair half, repairing interrupted tool-result ordering, whereas the retry-classifier half described in points 1 and 2 appears to be tracked nowhere. Repair alone would leave permanent failures being retried; a corrected classifier alone would leave the wedged transcript unrepaired. Both halves are needed for a wedged session to either recover or fail terminally.

Provenance: this report was first filed as issue #1490, which the intake automation closed as not_planned twelve seconds after submission under the vouched-reporter rule, before any human review. It is re-submitted here because Discussions is the intake channel the project documents for reports from unvouched reporters.

🤖 Generated with Claude Code