Feature request: declarative cross-model and cross-provider fallback chains

[sprintberlin]

Feature request

Add an opt-in, declarative fallback chain to CLIProxyAPI core so one client-facing model can fail over to explicitly configured alternative models and providers when the preferred route is temporarily unavailable or its quota is exhausted.

A practical example is a client requesting gpt-5.6-sol. If every eligible Codex credential for that model has exhausted its quota, CLIProxyAPI could transparently retry the same request with grok-4.5, provided the operator explicitly configured that fallback and the request has not started producing downstream output.

This is not a request for implicit fallback to arbitrary models. The chain should be explicit, ordered, opt-in, bounded, and compatibility-aware.

Why this is valuable in real deployments

CLIProxyAPI is particularly useful as a unified gateway across subscription-backed OAuth accounts, API-key providers, and OpenAI-compatible backends. In practice, these resources do not fail or reset at the same time:

• one model can reach a weekly or rolling usage limit while another provider remains fully available;
• one provider can return a temporary 429, 502, 503, 504, overload, or maintenance error;
• different credentials for the same model may all be unavailable even though another configured model could complete the task;
• operators currently need client-side switching, an additional proxy layer, or a custom plugin to preserve service continuity.

Credential rotation solves only part of this problem. Retrying ten credentials that all expose the same exhausted model still ends in a hard failure. A model fallback chain solves the next layer: after the eligible credential pool for a route is exhausted, continue with the next explicitly approved model/provider route.

In my opinion, this would be highly useful in production because the gateway already has the information and execution boundary needed to make the decision correctly. Every client implementing its own fallback logic duplicates configuration, error classification, retry limits, streaming safety, and observability. A core implementation would provide one predictable policy for every OpenAI-, Responses-, and Anthropic-compatible client using the gateway.

It would also make subscription-backed pools much more practical. Their limits are often model-specific and time-bucket-specific, so temporary exhaustion should not necessarily make the complete gateway unavailable.

Proposed configuration shape

The exact schema is open for discussion. One possible design:

model-fallbacks:
  gpt-5.6-sol:
    strategy: fill-first
    max-attempts: 3
    candidates:
      - provider: codex
        model: gpt-5.6-sol

      - provider: xai
        model: grok-4.5
        on:
          statuses: [429, 502, 503, 504]
          reasons:
            - quota_exceeded
            - usage_limit_reached
            - rate_limited
            - overloaded

      - provider: claude
        model: claude-opus-5
        on:
          statuses: [429, 502, 503, 504]
          reasons:
            - quota_exceeded
            - rate_limited
            - overloaded

Alternatively, a logical client-facing alias could own the chain:

model-fallbacks:
  resilient-coding:
    candidates:
      - provider: codex
        model: gpt-5.6-sol
      - provider: xai
        model: grok-4.5
      - provider: claude
        model: claude-opus-5

The client would always request resilient-coding, while CLIProxyAPI keeps the configured order and switches only under the configured conditions.

Suggested behavior

1. Exhaust credentials before changing the model

For each candidate, CLIProxyAPI should retain its existing credential selection, cooldown, retry, priority, round-robin, fill-first, and session-affinity behavior.

Only after no eligible credential can serve the current candidate, or after a configured terminal fallback condition is received, should execution move to the next candidate.

Conceptually:

requested logical model
  -> candidate 1
     -> credential A
     -> credential B
     -> no eligible credential / configured fallback error
  -> candidate 2
     -> credential C
  -> candidate 3

2. Explicit error classification

Fallback conditions should be configurable and should support both HTTP status codes and normalized provider reasons. Model-specific subscription errors are not always represented consistently as HTTP 429; examples include usage_limit_reached, quota exhaustion, account capacity errors, and provider overload responses.

The implementation should avoid fragile ad hoc string matching where possible. A normalized internal error classification would also improve logging and future provider support.

Fallback should normally not occur for semantic client errors such as invalid requests, unsupported tools, context overflow, or authentication failures unless an operator explicitly enables those conditions.

3. Safe streaming semantics

For streaming requests, switching is safe only before the first meaningful downstream chunk has been committed. Once tokens, tool calls, or other response items have been emitted, CLIProxyAPI should not splice a second model into the same response.

This behavior should be explicit and covered by tests for SSE and Responses/WebSocket paths where applicable.

4. Compatibility-aware candidates

A candidate should be skipped if it cannot satisfy the request protocol or required capabilities. Examples:

• /v1/responses support;
• tool/function calling;
• image or document input;
• structured output or required response format;
• required context size;
• server-side tools;
• model-specific reasoning or replay requirements.

An explicit per-candidate capability declaration could be supported if automatic detection is not reliable. This avoids falling back from a capable primary model to a backend that accepts basic chat but cannot complete an agent/tool loop.

5. Session behavior

The fallback result should integrate with session affinity. A successful fallback could either:

• remain sticky for the session until the candidate recovers or the affinity expires; or
• be retried from the primary candidate on every new request.

Both policies are useful, but they should be deterministic and configurable. A sticky fallback is generally preferable for multi-turn agent sessions because switching model families on every turn can affect reasoning style, tool behavior, prompt caching, and provider-specific continuation state.

6. Bounded retries and loop prevention

The chain should have a maximum number of attempts and must prevent recursive aliases or cycles such as A -> B -> A.

Provider retries, credential retries, and model fallback attempts should have a clearly defined combined budget so a single request cannot amplify into an unexpectedly large number of upstream calls.

7. Observability and transparency

Logs and usage events should record at least:

• requested client-facing model;
• selected candidate/provider/model;
• credential attempts, without exposing secrets;
• normalized reason for each fallback;
• final provider/model;
• whether the response was served by the primary route or a fallback.

If compatible with the existing API behavior, an optional response header or usage metadata field could expose the final resolved model. This matters for debugging, accounting, and evaluating differences in output quality.

Relationship to existing configuration

The current configuration already provides important building blocks:

• request-retry retries selected transient status codes;
• max-retry-credentials limits credential rotation;
• cooldown and quota handling mark credentials/models unavailable;
• routing.strategy controls credential selection;
• oauth-model-alias and per-auth model-aliases rename or remap OAuth models.

These features operate primarily at the retry, credential-selection, or alias-mapping layer. They do not provide a generic ordered chain that says: "try this provider/model first, then this different provider/model only if the first route is exhausted or unavailable."

A fallback chain should complement, not replace, the existing behavior.

Related work

This request builds on several previous discussions and implementation attempts:

• Feature request: per-auth OAuth model aliases or quota-aware model fallback #3764 requested per-auth OAuth aliases or quota-aware fallback, but focused on a narrower Codex OAuth scenario.
• Feature request: priority-based auto routing between API-key providers and Codex OAuth fallback #3964 discussed priority-based routing between API-key providers and Codex OAuth and the possibility of implementing it as a plugin.
• Feature Request: Fill-First Strategy for Alias Groups to Prevent Model Switching Within Sessions #3529 discussed fill-first behavior and consistency within alias groups.
• Codex requests may fail because Cliproxy falls back to incompatible providers/models #2704 demonstrated why fallback must be protocol/capability-aware, especially for Codex /v1/responses requests.
• PR feat: add global model aliases with cross-provider fallback #764 and PR feat: add global model aliases with cross-provider fallback #765 attempted global model aliases with cross-provider fallback on 429, 502, 503, and 504.
• PR fix: round-robin, fallback chains, cross-provider failover #1613 attempted configurable cross-provider chains, quota tracking, and reset-aware routing.
• Apparux/cpa-plugin-priority-auto-router demonstrates that the plugin APIs can implement prioritized candidates and pre-first-chunk fallback.
• The claude-web-search-router example in this repository also demonstrates an executor-managed fallback chain across multiple backends for a specialized use case.

Those efforts show both clear demand and technical feasibility. The distinction of this request is a small, generic, declarative, provider-neutral core policy rather than a provider-specific workaround or client-specific plugin.

Why core support in addition to plugins

A plugin remains a valid solution for custom routing. However, fallback is a foundational gateway concern when it combines native credential state, quota cooldowns, provider error normalization, session affinity, protocol translation, streaming commitment, and usage accounting.

Core support could:

• reuse the native auth and cooldown state without duplicating it;
• apply consistently across supported clients and protocols;
• provide standardized error classification and observability;
• avoid every operator maintaining a security-sensitive routing plugin;
• make the behavior discoverable and supportable through the normal YAML configuration;
• reduce the risk of recursion or double retries through host.model.* callbacks.

The implementation does not need to replace plugin routing. A minimal core primitive could also give plugins a safer common fallback engine.

Suggested initial scope

A manageable first version could include:

1. opt-in ordered fallback candidates per client-facing model;
2. cross-model and cross-provider routing;
3. fallback on a configurable set of normalized quota/transient errors;
4. exhaustion of eligible credentials before advancing to the next candidate;
5. a global/per-chain maximum attempt budget;
6. stream fallback only before the first downstream chunk;
7. cycle validation during configuration loading;
8. clear logs and usage metadata for the resolved route;
9. tests for non-streaming, SSE streaming, quota exhaustion, incompatible candidates, and retry limits.

More advanced capability matching and reset-aware recovery policies could follow later.

Acceptance examples

Quota exhaustion

client requests gpt-5.6-sol
-> Codex credential A: usage_limit_reached
-> Codex credential B: usage_limit_reached
-> chain advances to xAI/grok-4.5
-> request succeeds

Temporary outage

client requests resilient-coding
-> primary provider returns 503 before output
-> configured secondary provider succeeds

Streaming safety

primary provider returns 503 before first SSE event
-> fallback is allowed

primary provider emits a content delta and then fails
-> fallback is not allowed; terminate the stream with the appropriate error

Non-fallback client error

primary provider returns 400 because the request schema is invalid
-> no fallback by default
-> return the client error immediately

I believe this feature would materially improve CLIProxyAPI as a resilient multi-provider gateway while keeping routing behavior explicit and under operator control.