How do you configure OmniRoute for multi-provider failover?

[diegosouzapw]

I'd like to understand the best practices for setting up OmniRoute with multiple AI providers for automatic failover.

Questions:

1. What's the recommended way to configure priority between providers (e.g., OpenAI → Anthropic → Google)?
2. How does the health-check mechanism work to detect provider outages?
3. Are there any performance tips for reducing latency in the routing layer?

Any tips or examples from your setup would be greatly appreciated! 🚀

[minhdangnguyenn]

here are some suggestions :D
Priority Configuration:
List providers in your routing config by order (OpenAI first, then Anthropic, then Google). The router switches to the next if one fails.

Health Checks:
Automated checks (like ping or simple prompt) run every few seconds/minutes. If a provider doesn’t respond or errors, it’s temporarily skipped.

Latency Tips:
Cache health check results, keep connections warm, and avoid per-request checks. Only hit next provider if the first fails—don’t wait for long timeouts.

Practical Advice:
Monitor provider status, set reasonable failover thresholds, and observe usage/latency through metrics. Start with simple priority and only tweak as needed.

[diegosouzapw]

Yes, noted

[diegosouzapw]

Great question — here's the OmniRoute-native way to set this up:

1. Combos = your failover configuration
A Combo is a named group of provider accounts with a routing strategy. Go to Dashboard → Combos → New Combo and add your providers in priority order (OpenAI → Anthropic → Google). The combo engine will automatically route to the next provider when the current one fails.

2. Routing modes

• Failover — tries providers in order, switches on any error (429, 500, timeout)
• Load Balance — distributes requests across all healthy providers
• Auto — the Auto-Combo engine learns from latency/cost/reliability metrics and selects the best provider dynamically using a 6-factor scoring algorithm

3. How health detection works
OmniRoute tracks errors per account with exponential-backoff cooldowns (1s → 2min). A 429 triggers an automatic cooldown on that account — your IDE never sees it. Once the cooldown expires, the account is probe-admitted back gradually. You can see active circuit breakers in real time on /dashboard/health.

4. Latency tips

• Use Auto mode on your combo — the bandit exploration algorithm continuously balances between exploitation (known best) and exploration (probing alternatives)
• Enable the Semantic Cache to avoid re-fetching identical prompts
• Use best_combo_for_task via the MCP server to let an agent select the best combo for a given task type

The /dashboard/analytics page shows per-provider latency, cost, and reliability over time — useful for tuning your combo configuration.

[diegosouzapw]

Closing this thread as stale due to inactivity.

No maintainer follow-up happened after the last maintainer message for over 15 days. If this is still relevant, feel free to open a new discussion with updated logs/steps and I’ll pick it up with fresh context.

[faraz-jb]

Here's how we run multi-provider failover in production (290+ providers routed via OmniRoute):

1. Combo priority = failover order (exactly as Diego described)

• Dashboard → Combos → New Combo → add providers in priority order
• We use: OpenAI (primary) → Anthropic → Google → Groq → Together.ai → fallbacks
• Each provider account in the combo gets a weight — higher = more traffic when healthy

2. Health checks are per-provider-account, not per-provider-type

• OmniRoute pings each account's /models or a cheap /chat/completions every 30s
• If 3 consecutive checks fail → account marked unhealthy → traffic shifts to next in combo
• Critical: Set health_check_interval_seconds in your config (we use 30s); default 60s is too slow for production

3. Latency reduction tips that actually matter:

• Enable stream: true on all combos — streaming starts returning tokens before full response, cuts perceived latency 40-60%
• Use timeout_seconds per provider — we set 15s for OpenAI, 20s for Anthropic, 10s for Groq; prevents hanging on slow providers
• Combo-level max_retries: 1 — don't let OmniRoute retry internally; let your app handle fallback (cleaner observability)
• Prefer litellm format providers — they normalize request/response, avoid transformation overhead

4. Our production combo config snippet:

combos:
  - name: "primary-reasoning"
    strategy: "priority"
    providers:
      - name: "openai-gpt4o"
        weight: 100
        timeout_seconds: 15
      - name: "anthropic-claude-3.5-sonnet"
        weight: 80
        timeout_seconds: 20
      - name: "google-gemini-1.5-pro"
        weight: 60
        timeout_seconds: 15
    health_check_interval_seconds: 30
    max_retries: 1
    stream: true

5. Gotcha we hit: Free-tier providers (Groq, Together, OpenRouter free models) return 502/429 differently than paid — they often succeed on HTTP but return error in body. We added a response validator in our wrapper that treats { "error": { "code": "rate_limit" } } as a health-check failure so failover triggers automatically.

Happy to share our full docker-compose + Traefik labels if useful — we run this behind Traefik with TLS termination and it's been stable for months.