[Discussion] Quality-triggered escalation for model routing

[purushah]

Hi all. I'd like to propose letting a router escalate to a stronger candidate when the cheaper model's answer is not good enough, not only when the call fails. This is the "cascade" pattern. I want to agree on the shape here before opening a PR.

Today

A MODEL_ROUTER (#897) picks one candidate per request with rules, an LLM judge, or a custom executor. With fallback(true) the remaining candidates are tried in declaration order, but only after the selected model fails, after its retries or immediately for a pre-call failure such as an unresolved resource. The loop does check a response for truncation, a missing structured output and the like, but a successful answer is final however poor it is, and tool rounds stay on the selected model. In short: routing decides before the call, and nothing checks the quality of a successful answer after it.

How others do it

Pattern	Decides	Signal	Examples
Predictive routing	before the call	predicted difficulty or quality	Bedrock Intelligent Prompt Routing, RouteLLM, OpenRouter Auto, LangChain4j RoutingChatModel, Flink Agents' LLM judge (#897)
Cascade	after the call	quality of the actual answer	FrugalGPT and AutoMix (research); OpenRouter's confidence-threshold guide and its "Jev-verified cascade", where Jev is a decision model (recipes in application code); built in: Portkey, where a failed output guardrail becomes a status code that its fallback config acts on, and NVIDIA NeMo Switchyard, where an LLM judge rates the weak model's completed turn, the buffered reply is discarded and the strong model serves the same turn

Gateways (LiteLLM, OpenRouter, Portkey) declare the fallback list explicitly, cap it at a few models, and trigger on errors; Portkey's guardrail status codes are the one quality trigger. Every system above judges a complete answer, none a streamed one. #1324 proposes a LiteLLM Router integration for routing and failover among deployments; this proposal is about answer quality across models and is complementary.

Three findings shaped the design:

• Two tiers are enough. Bouchard (2026, v2): a second model adds 2.1 to 8.2 points over the best single model at matched budget, averaged across budgets; a third, fourth or fifth model gave no budget-averaged gain in 118 of 135 comparisons and at most 0.43 points.
• The signal must be about the answer, not the question. Same paper, simulated scores: a score that also encodes shared question difficulty removes the gain from a third tier even at the same AUROC, because the next model fails on the same questions; the two-model gain survives.
• Confidence is not calibrated out of the box. Self-scores run high. Decision models report a probability but are not calibrated everywhere: an independent evaluation of Jev measured expected calibration error of 0.011 on one task and 0.108 on another. Thresholds come from error rates on your own traffic.

Proposal

An optional escalation policy next to fallback. Fallback keeps handling errors. Escalation handles answers that pass but score badly.

ModelRouter.of("small", "big")
        .fallback(true)                                  // on failure: next candidate, as today
        .escalation(Escalation.tiers("small", "big")     // after the answer: move up?
                .when(QualitySignals.answerConfidence("confidence", 0.7)))
        .build();

Names are illustrative. This example sets no strategy(...), which is required today; I propose it becomes optional when tiers are set, so every request starts on the first tier. A strategy may still pick the starting tier. A low confidence, or a structural problem, re-sends the original request to big, with big's own prompt and tools; the cheap attempt is discarded. big is the last tier, so its answer is final.

Signals are checked after each model round and before any tool call from that round is dispatched, so a discarded attempt has no side effects. The structural signal is always on; when and or add signals to it. Responses the loop already treats as failures (truncation, content filter, missed output schema) stay on the fallback path, and failure wins when both apply.

1. Structural (always on): empty content with no tool call, a call to a tool the candidate does not have, arguments that fail the tool's input schema, the same tool call repeated with identical arguments in consecutive rounds. These are pure checks on the response: new code in the chat action, no model call, deterministic on replay.
2. Answer confidence (opt-in): the candidate's prompt asks for a numeric confidence field. When the request uses structured output the field is added to the schema and stripped from the returned content; otherwise it is read from a trailing JSON block. An ordering signal, not a probability. Exact mechanics are question 5.
3. Decision model on the answer (opt-in): a DECISION_MODEL ([api][java][python] Add DecisionModel resources and Event/Action calls #1264) gets the request plus the answer and one predicate question; probability_true below the threshold escalates. A direct call under durableExecute with id escalate:<router>:<candidate>, not through the decision action, so ids do not collide. On tool rounds the route is direct, so the router name comes from the routing metadata parked for the request. Picking the starting tier with a decision model is [Feature] Decision-model routing strategy for MODEL_ROUTER #1325.
4. Custom: an executor with the request and response, like CustomRoutingExecutor.

Tiers are explicit, cheapest first, at least two, each a candidate; candidates left out stay fallback-only. Tier count bounds the hops, maxEscalations(n) can cap lower. Two tiers, one hop, is the recommended default. The last tier always accepts on quality. If it fails after its retries, I propose fallback skips candidates that already answered this request, which is a change from today's fixed order; what happens with none left is question 6. Declaration order in of(...) keeps meaning fallback order.

Tool loop: I propose that once a request escalates it stays on the stronger model (question 3). Switching back per round breaks prefix caching. Today tool rounds run under the plain chat id with no fallback; that stays as it is.

Replay: each initial attempt already has its own durable id, chat:<router>:<candidate>. Confidence and decision-model verdicts are recorded beside it; structural checks are recomputed.

Observability and cost: a ModelRoutingEvent with source escalation, the signal and score. In the model_routing block, decision_source records the last step that changed the model, and escalation_attempted is kept next to fallback_attempted, with the discarded attempt's tokens. Today fallback_attempted is derived from the final model differing from the selected one; with escalation it has to come from an actual fallback attempt. Two metrics next to routingDecisionLatencyMs: routingEscalationCount and routingEscalatedTokens. An escalated request pays the cheap call, the strong call and any judge or decision call, so the cascade costs more than calling the strong model directly once the escalation rate exceeds roughly (big minus small minus judge) over big. The docs should show that, and #1065, the per-agent cost budget, is where a ceiling belongs.

Scope: Java first. Python follows the parity work in #1062; registering a MODEL_ROUTER from Python raises today and the routing event is Java-only.

Questions

1. Separate escalation policy with its own tiers, or widen fallback with a quality trigger and reuse declaration order? I lean separate, so the error path does not change.
2. Should answer confidence ship built in, or only as a custom signal with an example? I lean built in, with the calibration caveat in the docs.
3. Sticky escalation for the whole request, or per-round inside the tool loop?
4. Should strategy(...) become optional when tiers are set?
5. For answer confidence, is adding a field to the caller's structured-output schema acceptable, or should it be a separate signal call?
6. When the last tier fails, fail the request, or return the discarded cheaper answer flagged escalation_failed? I lean fail, so callers never get an answer the policy rejected.

References: OpenRouter: confidence thresholds, OpenRouter: Jev-verified cascade, Vals AI: Jev evaluation, Bedrock Intelligent Prompt Routing, LangChain4j model router, Portkey guardrails, NVIDIA NeMo Switchyard escalation router, FrugalGPT, AutoMix, Bouchard, Is escalation worth it?, Gupta et al., LM cascades.

[wenjin272]

Thanks for the proposal. My preferences on the six questions are:

1. Keep escalation separate from fallback. They address different scenarios: fallback handles invocation failures, while escalation handles successful responses with insufficient quality. Users should be able to configure them independently.
2. Start with a custom quality-signal interface and examples. Built-in answer confidence would require changes to prompts, output schemas, and response parsing. I would defer that until we have more experience with concrete use cases.
3. Keep escalation sticky for the request. Once a request moves to a stronger model, subsequent tool rounds should continue using it.
4. Allow strategy(...) to be omitted when tiers are configured, defaulting to the first tier.
5. Defer the schema-modification question, since we would start with custom signals rather than built-in answer confidence.
6. Fail the request when retries and any applicable fallback options are exhausted, rather than returning an answer that the quality policy already rejected.

I would also appreciate some clarification on two aspects:

• What exactly are we evaluating, and how is “good enough” defined? Structural validity, a model’s self-reported confidence, and whether an answer is correct or satisfies the request are different signals. What criteria would the evaluator use, and which parts would users need to define? It would help to walk through concrete examples for both a completed answer and a response containing only tool calls, including the evaluator’s inputs, the acceptance criteria, and how the result triggers escalation.
• What happens to prior progress when escalation occurs mid-request? Does the stronger model receive the existing conversation and tool results, or restart from the original request? Checking before dispatching the current round’s tools prevents side effects from that round, but earlier rounds may already have produced side effects. How would those results be preserved and repeated execution of those operations avoided?

[purushah]

Thanks. Agreed on all six. One adjustment on 2: I'd keep the structural checks built in, since they need no prompt or schema change, and ship a custom signal interface with one example that uses a decision model on the answer. Built-in answer confidence and the schema question wait.

What is evaluated. The evaluator gets the request, the candidate's response (content and tool calls) and the candidate's tool definitions, and returns accept or escalate with a reason and an optional score. Structural checks are always on: empty content with no tool call, a call to a tool the candidate does not have, arguments that fail the tool's schema, the same call repeated with identical arguments. Custom signals and the decision-model example are thresholds the user sets. "Good enough" means every configured signal accepts; the framework does not judge correctness itself.

Completed answer: tiers can be two or three models, cheapest first. With small, medium, big: small returns SQL text, structural passes, the decision model scores 0.55 against 0.7, so medium gets the original request. medium scores 0.65, still below 0.7, so big gets it and, as the last tier, its answer is final. Each hop pays for the discarded attempt, so two tiers is the recommended default and three is for when the middle model catches most escalations.

Tool-call-only response: structural checks the tool exists for small and the arguments validate. If that fails we escalate before dispatching, so the bad call has no side effects. The decision-model signal skips tool-only rounds by default. Tool-only rounds are where escalation has the least to say; the quality signal applies to the answer the model writes after the tools return.

Prior progress. The stronger model continues the conversation, it does not restart. It gets the original request plus every completed round with tool results; only the rejected round is dropped, and its tools never ran. Earlier tool results stay in the conversation and in the journal under tool-call:<id>, so nothing is re-executed. A restart would discard that work and is the path that could repeat side effects.

Next from me: a short write-up with the signal interface, the example and these walk-throughs, then the PR. The starting tier can come from #1330.