More than 8 images per request wedges the session: 400 is never retried and images are never dropped

[ThomasLHQ]

Summary

llm-pi-ai bounds request images by total bytes (maxRequestImageBytes, default 20 MiB) but not by count. Backends that cap images per prompt (DeepSeek engines: 8) reject the request with HTTP 400 At most 8 image(s) may be provided in one prompt. That 400 is not retryable, and no fallback model is configured, so the agent loop re-sends the same over-limit context on the next turn. The session can never make progress; every subsequent turn repeats the identical error.

Reproduction

1. Configure an openai-completions provider backed by a DeepSeek engine (8 images per prompt).
2. Attach more than 8 images across the conversation — old images in the history count too.
3. Send any message. The request fails with 400, and the next turn sends the same context and fails again.

Current behavior

The image offload trigger only passes a byte bound:

• packages/llm/llm-pi-ai/src/adapter.ts — requestImagePolicy carries maxRequestImageBytes only.
• packages/llm/llm-pi-ai/src/config.ts — maxRequestImageBytes schema and default; no maxImages.

Expected behavior

Bound the count the same way as bytes. When a request would exceed the limit, drop the oldest images (keep them in the conversation as text placeholders) and continue the turn. We run maxImages: 8 in production as a patch on the same offload path, applied to 0.2.1-rc.2. Alternatively, treat the backend's per-prompt image-limit 400 as retryable after trimming. Happy to send a PR for the count bound.

Environment

• dsh 0.2.1-rc.2
• Provider: openai-completions via LiteLLM, engine enforcing an 8-images-per-prompt limit

[PerryLink]

Confirmed on 0.2.1-rc.2, with one correction that makes your patch smaller than you think and one that makes the alternative worse.

The count is genuinely unbounded. packages/llm/llm-pi-ai/src/config.ts:167-173:

/**
 * Maximum base64-encoded image payload per request. When a request's
 * accumulated images exceed it, the oldest images are replaced by text
 * placeholders until the request fits, so a long session keeps completing
 * requests instead of being rejected by a request-size cap.
 */
maxRequestImageBytes?: number

and the policy actually handed to the converter, packages/llm/llm-pi-ai/src/adapter.ts:410-415:

maxRequestImageBytes: profile.maxRequestImageBytes,
requestImagePolicy: {
  maxPixels: profile.requestImagePixelBudget,
  maxBytes: profile.requestImageMaxBytes,
},

Three bounds, all of them size-shaped — per-request bytes, per-image pixels, per-image bytes. Nothing counts images.

The part worth leading with: the offload loop you want already exists. That doc comment is describing implemented behaviour, not an aspiration — when the byte bound is crossed the oldest images are already replaced by text placeholders. So a count bound is a second predicate inside a loop that is already trimming oldest-first, at the single site adapter.ts:407 (toPiContext). That is a much easier PR to argue for than new machinery, and it also means your two bounds should share the same ordering, or they will fight over which image goes next.

On the alternative: treating the 400 as retryable would contradict the documented intent. packages/llm/llm-pi-ai/src/stream.ts:48-49 maps 413/400 to INVALID_REQUEST with the comment "resending the same request cannot succeed, so it is invalid, not transient", and packages/llm/llm/src/retry-policy.ts:18-24 is an allowlist that does not include it. Retrying the identical over-limit context is exactly the behaviour that comment forbids — trimming is the shape that fits.

One thing to say out loud in the report: the 8-image ceiling belongs to the backend, not to DSH, and DSH has no way to discover it. So a fixed maxImages: 8 default is a guess that will be wrong in both directions. If you propose the field, propose it as provider-configurable alongside maxRequestImageBytes, with the drop-oldest behaviour as the default response when it is exceeded — and say what happens when the user's newest images are the ones that get dropped, because that is the case someone will push back on.

[zklall]

Same lock-up on 0.2.1-rc.3, reached through a different protocol adapter (api: openai-responses) and a different backend ceiling (30 images/request). Three things worth adding, one of which touches the "don't retry the 400" argument.

What was observed

An llm-pi-ai provider (openai-responses) whose gateway enforces 30 images per request. After a session accumulated 31 images, the turn failed with

invalid_request_error: Image count 31 exceeds limit 30 per request.
code: INVALID_REQUEST

Turn 1 failed at step 57 (the step that pushed the history to 31); the next two turns failed at step 1 with the identical message. Failed turns contribute nothing to history, so the same 31 images are re-sent forever — the session is dead, and retrying cannot help. Same end state as the report above, independent of protocol.

The byte bound was nowhere near firing

At failure the session held 32 image blocks, ~14.2 MiB of base64 total — under the 20 MiB default maxRequestImageBytes. So the existing oldest-first trim could not have started; the only bound that would have saved this session is a count bound.

This is a second data point for leading with count: on screenshot/PNG-heavy sessions the byte bound is a poor proxy. Here the average was ~450 KB of base64 per image, so the 20 MiB default tolerates ~45 images — 1.5× past a 30-image cap, and ~5.6× past the 8-image cap in the original report. The bound is off in both directions depending on image shapes, which is the same argument as making it provider-configurable.

"Trim then retry" is not "retry the 400"

I am not proposing to make INVALID_REQUEST retryable, so this does not conflict with stream.ts:48-49. The trim produces a different request, which is what IMAGE_OFFLOAD_REQUIRED already expresses. Two independent paths:

1. A provider-configurable count bound next to maxRequestImageBytes, sharing the same oldest-first trim loop at toPiContext (as proposed above).
2. For sessions already over a cap DSH cannot know: classify a recognizable count-clamp rejection (Image count N exceeds limit M, At most N image(s) may be provided in one prompt) as IMAGE_OFFLOAD_REQUIRED carrying offloadImages = N - M, rather than a terminal INVALID_REQUEST. The retry allowlist stays untouched, and the result is one-shot trim-then-retry rather than a blind resend.

Workaround that works today, no patch

Lower the provider's maxRequestImageBytes until the byte bound fires before the backend count cap:

- id: llm-pi-ai
  name: "@deepseek-ai/dsh-llm-pi-ai"
  config:
    providers:
      <provider>:
        maxRequestImageBytes: 8388608   # 8 MiB ≈ 17 of my ~450 KB images

Caveats worth documenting if this becomes advice: (a) it is a proxy — many small images (say 40 KB each) fit 8 MiB at ~200 images and will still wedge the session; (b) it silently drops older visuals for every session on that provider, and the placeholders only keep a path that has to be re-read.

The missing escape hatch

Once wedged, nothing in the product recovers the session — there is no trim/compact path for a session that cannot complete a request. Even a targeted recovery ("the last turn failed on an image-count cap; re-run with the N oldest images offloaded") would be enough to unstick people, and it would also cover every unknown backend ceiling.

[hoveychen]

Another backend hitting the same wedge on 0.2.1-rc.2: an openai-completions route (api.abliteration.ai) caps requests at 4 images. The 5th image read in a turn ends it with

400 {"code":"media_count_exceeded","message":"This request has more images than the limit of 4. ...","type":"media_error"}
code: INVALID_REQUEST

and every follow-up re-sends the same history and fails the same way.

The core already supports a count bound — requiredImageOffload / offloadedImagePrefixCount honour budget.maxImages, and llm-deepseek passes maxImagesPerRequest — so llm-pi-ai only needs to forward one. Locally I added a per-provider maxRequestImages (validated like maxRequestImageBytes, carried on the profile) and passed it as maxImages next to maxBytes in toPiContextWithImages. With maxRequestImages: 4 the wedged session resumed: IMAGE_OFFLOAD_REQUIRED → image/offload of the oldest occurrence → retry succeeded, and later reads kept offloading as expected.

+1 for shipping a count bound on llm-pi-ai provider config.

[PerryLink]

Checked on master ce7073d5. The core side is already there, and the gap is exactly the line you named.

• packages/llm/llm/src/content.ts:285-289 — offloadedImagePrefixCount derives excessCount from budget.maxImages when it is defined; packages/llm/llm/src/types.ts:406 declares maxImages?: number on the budget.
• packages/llm/llm-deepseek/src/images.ts:17 passes maxImages: connection.maxImagesPerRequest into bounds(), used at :82. So one adapter uses the count bound end to end.
• packages/llm/llm-pi-ai/src/context.ts:360-371 — the only requiredImageOffload call in that package passes { representation: 'base64', maxBytes: maxRequestImageBytes }. No maxImages. Your local patch is the right shape.

One trap for whoever lands it: that whole block sits inside if (maxRequestImageBytes !== undefined). A provider configured with maxRequestImages but no byte bound would skip the block entirely, so the guard has to become "either bound configured" — otherwise forwarding the field alone is a no-op for exactly the 30-image and 4-image gateways you two reported, where the byte bound was nowhere near firing.

Both datapoints are worth keeping in the PR: openai-responses at 30 and openai-completions at 4 show the ceiling is provider-owned, so the value has to be configurably forwarded rather than derived from a protocol.