Provider fallback for same-format inference endpoints

[usize]

Summary

Praxis has no provider fallback mechanism. When a primary inference provider is unavailable, requests fail rather than falling through to an alternative endpoint serving the same API format.

The simpler case - same API format, different endpoint, different credentials, optional model name swap - is still worth pursuing since it covers the core use case of failing over to a different backend while requesting a model with the same protocol.

This covers local-first setups (vLLM primary, cloud fallback), multi-provider redundancy, and cost tiering (free tier primary, paid fallback).

An experimental prototype exists on an experiment/fallback-filter branch that validates the approach using CalloutClient for outbound requests to an ordered target list. It works for non-streaming requests today. However, this prototype reveals structural limitations in Praxis as it currently exists. Solving those will allow us to create a better fallback experience that supports streaming.

The problem

AI gateways universally bundle backend, credentials, and model into a single object because fallback requires atomic substitution of all three. Praxis splits these across orthogonal primitives: router selects clusters and credential_injection attaches auth, while model name swapping is handled by the generic json_body_field filter rather than a dedicated primitive. This separation of concerns is clean for single-provider pipelines but creates complexity for fallback since a failed target requires coordinated rollback of cluster selection, credential injection, and model rewriting.

A provider_fallback filter sidesteps this by owning the full request lifecycle for its targets. Each target is a self-contained provider definition (URL, headers, model override, timeout) with its own circuit breaker.

A possible approach

The prototype uses CalloutClient (reqwest) for outbound requests, which works but cannot stream responses and bypasses Pingora's connection pooling and HTTP/2 support. Moving to production means using Pingora's native upstream dispatch instead, so that fallback targets get connection pooling, HTTP/2, TLS, backpressure, and streaming for free.

The core idea: the filter pre-bundles complete upstream state for each fallback target during the request phase (address, TLS config, credentials, model-rewritten body). The filter itself does not make outbound requests. Instead, it emits a plan that the Pingora proxy lifecycle consumes. On a retriable response status (5xx, 429), intercepted before response headers are committed downstream, the proxy reads the next target from the plan and retries through Pingora's existing retry loop.

This requires changes to the Pingora ProxyHttp handler hooks because the existing retry path only covers connection failures for idempotent requests. Provider fallback needs response-status-based retry for non-idempotent inference POSTs, with per-target credential and body swapping on each attempt. Without these changes, the filter cannot trigger a retry after receiving a retriable status, and streaming responses remain impossible since CalloutClient buffers the full response before returning it.

CalloutClient (reqwest) remains unchanged as a separate primitive with its own connection pool for side-channel callouts (guardrails, auth endpoints) where isolation from the provider connection pool is a security property.

Other alternatives

An alternative is to make fallback targets a proxy-native data structure that existing filters (credential_injection, router) plus model name swapping via json_body_field each populate independently. This would preserve the separation of concerns but introduces coordination problems: each filter needs to iterate over N targets, partial failures leave the target set in an inconsistent state (credentials injected for a target whose protocol translation failed), and the contract for when the target set is "complete" becomes implicit and fragile.

The bundled approach is more practical. A single filter assembles each target atomically (URL, credentials, model override, body transformation) and handles per-target preparation failures internally. The pipeline invariant is preserved: every filter executes once, forward-only. No cross-filter rollback is needed.

Summary

Other AI gateways have independently arrived at the conclusion that bundling credential injection, API translation, and model mapping into a single object works well for providing fallback capabilities. We should explore this and implement fallback capabilities likewise.

Our goal is to support streaming provider fallback, only for cases where no bytes have been committed downstream. Mid-stream fallback is an experimental feature prone to bugs in the gateways that attempt it, and something we can explore later in special circumstances, for example mid-stream fallbacks for backends that share a common engine like vLLM where mid-stream recovery might have more information to act on at the data plane layer.

Related

• Epic: Epic: Inference Provider Fallback ai#249
• HTTP callout filter: #358
• Sub-request orchestration: Proposal: New orchestration primitive, sub-requests from within the filter pipeline #87
• Provider failover with API translation: #281
• Unified AI types: #213

[shaneutt]

Thanks for the write-up @usize 🫡

The problem analysis is right on, the prototype validates it. I have thoughts on how we resolve it though:

What about using existing branch chains for fallback?

I think we should evaluate using existing features: branch chains + named chain references here rather than a new fallback filter. You wire up the Anthropic fallback chain once (credentials, translation, router), reference it by name from any branch condition, and you're done. The big win is composability: operators can plug custom filters into the fallback path, which gets hard with a monolithic filter owning the life-cycle. The coordination problem you called out is real, but it's scoped to one named chain definition, and we can catch misconfiguration at config validation time rather than discovering them during an actual fail-over.

If the YAML gets verbose (it will), I would like to consider solving that with higher-level abstractions that expand into the right branch topology than flatten the machinery to make config shorter.

The major blocker here

Yep, response-status retry doesn't exist in Pingora today, and that's the gap regardless of which approach we pick. However, I tend to see that as a deficiency in the server/protocol implementation in Pingora. I went search and there's an upstream PR that I think nails what we need here right on?

cloudflare/pingora#872

It adds an upstream_response_decision hook (48 lines, CI green) that fires after upstream headers arrive but before anything reaches the client. Returns an error with set_retry(true) and the existing retry loop handles the rest. I would like to cherry-pick this into the fork and wire upstream_peer to advance through a ranked target list on retry, and then advocate for this PR in the upstream. LMKWYT?

Massive payloads

The retry buffer is 64 KiB, which inference payloads will blow past. We still need to finish getting that configurable in fork+upstream. That constraint hits us everywhere.

[usize]

The branch-chain approach separates concerns cleanly, but, I get stuck at per-target state.

wire up the Anthropic fallback chain once (credentials, translation, router), reference it by name from any branch condition, and you're done.

Flipping your example: say Anthropic is primary, and we fall back to a trust-domain pool that authenticates via mTLS (no Authorization header).

credential_injection ran during request_filter and added an Authorization header to the request context.

On retry, upstream_peer runs again but the filter pipeline doesn't.

The headers and body mutations from the first pass are still on the context, so the Anthropic API is leaked to the wrong provider.

Branch chains fire during the pipeline pass that already ran, so there's no second pass where a "strip the Authorization header" branch could execute for the new target.

How do you see that being handled?

---

Edit

A possible middle ground:

A fallback filter which pre-computes targets via filter chains.

So we maintain the composability and clean separation of concerns--separate filter for credential injection, model selection etc... while also guaranteeing that we won't accidentally mix state between targets. Does that make sense?

An LLM generated ascii diagram ✨

              ┌─ chain A: [credential_injection] → [model_rewrite]
              │    bundle[0] = { anthropic.com, Bearer sk-ant, "claude-3-5" }
request ──────┤
              └─ chain B: [mTLS setup] → [model_rewrite]
                  bundle[1] = { mtls-pool, (no auth), "llama-3" }

ctx.extensions = [bundle[0], bundle[1]]
    │
    ▼
upstream_peer reads bundle[0] → anthropic.com → 503
    │
    ▼ (retry — advance index)
upstream_peer reads bundle[1] → mtls-pool.internal ✓
    ↳ fresh state, no leaked headers

This seems like a pattern that might even appear elsewhere. Where we have some chain of dependent mutations that can't be cleanly rolled back. So instead we pre-compute all possible states and apply them wholesale.

This is roughly the way databases handle transactions. The downside of it is, we introduce more latency with each fallback provider added.

[shaneutt]

Our proposal process has been updated to no longer require discussions so we'll close this: Instead create a feature request on the repo in question and maintainers will assess 🖖