Bug: context-window-exceeded errors aren't recovered when the provider is self-hosted vLLM

[vecchiotom]

Bug: context-window-exceeded errors aren't recovered when the provider is self-hosted vLLM

Environment

• dsh commit: b150a55 (confirmed unchanged through the current dsh@0.1.2-alpha.1 release tip)
• Provider: self-hosted vLLM, OpenAI-compatible endpoint (llm-deepseek adapter)

Symptom

Turn fails outright instead of auto-compacting and retrying:

This turn failed400: {"message":"This model's maximum context length is 204800 tokens. However, you requested 32768 output tokens and your prompt contains at least 172033 input tokens, for a total of at least 204801 tokens. Please reduce the length of the input prompt or the number of requested output tokens. (parameter=input_tokens, value=172033)","type":"BadRequestError","param":"input_tokens","code":400}

Happens intermittently — only on this specific error, not on other 400s from the same backend.

Root cause

dsh already has a working recovery path for exactly this case: packages/compaction/compaction-basic/src/index.ts listens for agent/request-error, checks failure.code === CONTEXT_WINDOW_EXCEEDED_CODE, and if so runs a compaction + retry (bounded by maxOverflowRetries).

That code is only assigned in packages/llm/llm-deepseek/src/adapter.ts, via httpErrorCode():

export function httpErrorCode(status: number, error?: WireError['error']): string {
  ...
  if (status === 400) {
    if (isContextWindowExceededError(detail)) return CONTEXT_WINDOW_EXCEEDED_CODE
    return 'INVALID_REQUEST'
  }
  ...
}

detail is built from providerError?.code/type/message, and providerError comes from:

const parsed = JSON.parse(rawResponse) as WireError
providerError = parsed.error

WireError in packages/llm/llm-deepseek/src/types.ts only models the OpenAI-nested shape:

export interface WireError {
  error?: { message?: string; type?: string; code?: string }
}

vLLM's OpenAI-compatible server does not nest its error body under "error" — it returns the fields flat at the top level ({"object":"error","message":...,"type":...,"param":...,"code":...}). This is a known, deliberate vLLM behavior — see [vllm-project/vllm#12886](vllm-project/vllm#12886), closed as "not planned," and [vllm-project/vllm#4667](vllm-project/vllm#4667) for the same flat shape in the wild.

So against vLLM, parsed.error is always undefined, detail is '', isContextWindowExceededError('') is false, and every 400 from vLLM gets classified as generic INVALID_REQUEST — including context-length overflows. The existing compaction-and-retry recovery in compaction-basic never fires, and the raw provider error surfaces to the user instead.

Checked git log b150a55..HEAD on adapter.ts, types.ts, and error.ts in packages/llm/llm-deepseek and packages/llm/llm: no relevant changes. The gap is present at b150a55 and unchanged at the current release tip.

Suggested fix

Accept both shapes when parsing the error body, e.g. in adapter.ts:

const parsed = JSON.parse(rawResponse) as WireError & { message?: string; type?: string; code?: string; param?: string }
providerError = parsed.error ?? (parsed.message !== undefined ? parsed : undefined)

and widen WireError in types.ts to also allow the flat shape (or normalize both into one shape at the parse boundary). Happy to open a PR if that's the preferred direction — wanted to flag the root cause first in case there's a reason the nested-only assumption was intentional.

Repro

Point llm-deepseek at a self-hosted vLLM instance, send a request that exceeds the served model's context window (prompt + max_tokens > --max-model-len), and the 400 will not trigger compaction/retry — it'll surface directly.

[argszero]

Confirmed source-verified against the current tip cd5ef81481 (0.1.2-alpha.1) — your root cause is exact, and I'd argue it's broader than the compaction/retry path you focused on.

The dropped detail breaks every error-classification consumer in this adapter, not just compaction.

The parse site (packages/llm/llm-deepseek/src/adapter.ts:663-669) builds detail from providerError:

providerError = parsed.error
const detail = [providerError?.code, providerError?.type, providerError?.message]
  .filter((field): field is string => typeof field === 'string')
  .join(' ')

That same detail is then consumed in three different places, all of which silently no-op against a flat vLLM body:

• :672 — providerRejectedFileId(detail) gates the stale-file invalidation + retry branch.
• :682 — providerRejectedNormalizedImage(detail) gates the normalized-image diagnostic branch.
• httpErrorCode() (:332) — gates the context-window classification that feeds compaction-basic.

Against vLLM's flat body ({"object":"error","message":...,"type":...,"code":...}), parsed.error is undefined → detail is ''. So all three consumers classify as "no signal." You correctly trace the context-window one; the stale-file and normalized-image branches are equally dead. This is a single parse-boundary bug with three downstream symptoms, which is why I'd fix it at the parse boundary (normalize flat → nested before it fans out), exactly as your suggested diff does, rather than patching each consumer.

One thing that makes the fix trivially safe: the classifier is already ready to fire on vLLM's wording. isContextWindowExceededError in packages/llm/llm/src/error.ts has a max context length branch:

|| /\b(?:maximum|max)(?:\s+(?:allowed|supported))?\s+context\s+(?:length|window)\b/i.test(detail)

Your sample message ("This model's maximum context length is 204800 tokens...") matches that branch. So the moment detail actually contains the message, isContextWindowExceededError returns true, httpErrorCode returns CONTEXT_WINDOW_EXCEEDED_CODE, and the existing compaction-basic retry path kicks in with zero further changes. No new classifier needed — the only bug is that the message is dropped before it reaches the classifier. That makes this a low-risk, high-value fix.

Also confirms a wider pattern: this is the second family member where a provider error body is parsed with a shape assumption (nested-only) that a real backend doesn't honor. (#4922 covers the attachment/TRANSPORT generalization gap.) Worth normalizing the shape at the parse boundary here and treating "flat vs nested WireError" as the rule rather than per-adapter special-casing.

Rationale for keeping the nested shape (so I understand the intent): is OpenAI's nested {"error":{...}} the contract the adapter is built against, and the flat shape a vLLM divergence this adapter never claimed to support? If so, the flat-shape tolerance is a self-hosted-compat improvement, not a regression — and it only adds signal where there was none before (detail was empty either way), so it can't misclassify a previously-correct case.

[dsc-tommaso-celano]

Good catch on the other two consumers, makes the case for fixing at the parse boundary rather than patching httpErrorCode() alone.

On your question: yes, nested is the intentional contract (it's the OpenAI wire format the adapter was built for), and vLLM's flat body is a deliberate divergence on their end (per the linked vllm issues, not planned to change). So this is additive compat, not a behavior change for OpenAI-shaped responses, nested bodies parse exactly as before, we're just no longer silently dropping the flat ones.

The fix is small, normalize at the parse boundary (parsed.error ?? (parsed.message !== undefined ? parsed : undefined), or a shared normalizeWireError() helper if you'd rather it be reusable for #4922) and widen WireError to accept both shapes. Should be a quick pickup for anyone with write access.

Happy to review or sanity-check a diff if that helps

[denial123789]

Added an Agent-first runbook for this boundary: self-hosted vLLM returns flat top-level error JSON, while the rc.2 adapter reads only parsed.error, so CONTEXT_WINDOW_EXCEEDED never reaches compaction-basic. The guide covers sanitized shape capture, normalized-code checks, and adapter tests for flat generic/context errors. https://github.com/sandbaseai/deepseek-harness-handbook/releases/tag/v0.5.388