Workers AI binding: no pre-dispatch request-size guard and no structural (status-based) overflow classification

[dknecht]

Summary

Follow-up / bigger-picture companion to #428. Even with #428 fixed, overflow handling on the Cloudflare Workers AI binding path is entirely reactive and message-based: the runtime dispatches the full projected conversation to ai.run(), waits for the provider to reject it, and then tries to recognize the rejection by regex over the error message (pi-ai isContextOverflow). Two structural gaps:

1. No pre-dispatch size guard. Nothing between the conversation projection and ai.run() bounds the serialized request. Proactive compaction (shouldCompact) is token-estimate-based against contextWindow, but the binding/gateway path can reject on request byte size (HTTP 413) well below the token threshold — e.g. large tool results accumulate bytes faster than token estimates predict. Those sessions never reach the proactive compaction threshold and only survive if reactive recovery classifies correctly.

2. Classification relies on message regexes when structured data exists. On the binding path the runtime has response.status (and puts it in CloudflareAIBindingError.meta.status), but overflow detection still round-trips through the flattened errorMessage string. A 413 from the binding is definitionally a request-size overflow and could be classified structurally, immune to provider message-format drift.

Why it matters

For long-lived persistent agents this failure mode is high-impact: the session's canonical conversation keeps growing across turns, and once it crosses the provider's request-size limit, every dispatch fails until something compacts the conversation. In our production deployment the practical effect was a steady stream of failed agent turns on the largest/oldest instances (75+ failures across ~12 instances in 3 days) — the instances most likely to be mid-task are exactly the ones that stop producing output.

Asks

1. Pre-dispatch guard: measure the serialized payload before ai.run() (byte size, plus the existing token estimate) and trigger the existing compaction pass proactively when it exceeds a configurable budget, instead of burning a failed provider round-trip.
2. Structural overflow classification: on the binding path, treat HTTP 413 (and provider bodies matching known context-length shapes) as overflow using the structured status already carried on the error, rather than relying on errorMessage regexes. This could live in the runtime's overflow check (session.ts:2585) as an OR with the pi-ai message-based check.
3. Consider surfacing a distinct error type (e.g. provider_request_too_large) so hosts can alert on "conversation outgrew the provider limit" separately from generic binding failures.

Environment: vendored @flue/runtime built from 4052d05e, Workers AI binding via AI Gateway, model @cf/moonshotai/kimi-k2.7-code (262,144-token window).

[FredKSchott]

Two of the three asks are addressed in the next release:

• Structural classification: overflow classification no longer depends on message-regex luck. A runtime-side classifier fronts both recovery sites (the live compact-and-retry loop and durable resume) and matches the runtime's own 413 marker — the code that writes the marker and the code that reads it share one constant — before falling back to the provider-pattern matching. That also removes the edge where a genuine 413 whose provider body mentions rate limiting could be vetoed by the generic non-overflow patterns.
• Distinct signal: CloudflareAIBindingError carries meta: { reason: 'request_too_large' } on 413 (additive — the error type is unchanged, so existing classifiers keep working). Since error meta reaches operation/log events, trackers can separate "conversation outgrew the provider limit" from a real binding outage structurally instead of parsing prose.

The discussion: the pre-dispatch size guard. It's real design surface — where to measure serialized bytes, how it's configured, and how it interacts with the token-estimate compaction threshold — and it also absorbs the "propagate structured provider-error info through agent-loop state" thread from #431. Proposals welcome here.