OpenBalancer

Aggregate multiple AI providers into a single OpenAI-compatible endpoint.

OpenBalancer is an open-source AI inference orchestration platform that unifies cloud and self-hosted LLM providers behind a single API. It provides intelligent routing, automatic failover, provider health monitoring, multi-user credential management, and an OpenAI-compatible interface while allowing users to securely bring their own provider API keys.

🌐 Live Demo 📄 Whitepaper

---

Dashboard

Dashboard Overview

OpenBalancer includes a built-in web dashboard for managing providers, credentials, API keys, and available models.

Features include:

• User authentication
• Secure provider credential management
• OpenBalancer API key generation
• Provider health monitoring
• Available model browser
• Language-specific SDK quickstarts
• Multi-user isolation

---

Why OpenBalancer?

Modern AI applications rarely rely on a single provider.

Developers often juggle:

• Groq
• OpenRouter
• Gemini
• Hugging Face
• Cerebras
• TogetherAI
• Ollama
• vLLM

Each provider has different:

• Rate limits
• Pricing
• Latency
• Available models
• Reliability

OpenBalancer abstracts these differences away, allowing applications to interact with a single OpenAI-compatible endpoint while intelligently routing requests across multiple providers.

Why shouldn't I just use Groq directly?

Configuration	Successful	Failed	Success Rate
Groq	140	360	28.0%
OpenRouter	254	246	50.8%
OpenBalancer	291	209	58.2%

Provider	Requests
OpenRouter	139
Groq	110
Hugging Face	30
Cerebras	12

What differentiates OpenBalancer from other providers?

Feature	Direct Provider	OpenBalancer
OpenAI-compatible API	✅	✅
Automatic failover	❌	✅
Multi-provider routing	❌	✅
Provider health monitoring	❌	✅
Routing policies	❌	✅
Multi-user support	❌	✅
Provider credential management	❌	✅
Dashboard	❌	✅
Vendor lock-in	High	None

---

Architecture

Architecture Diagram

Incoming Request
        │
        ▼
Authenticate User
        │
        ▼
Load User Provider Keys
        │
        ▼
Discover Healthy Providers
        │
        ▼
Apply Routing Policy
        │
        ▼
Try Selected Provider
        │
        │ Success?
   ┌────┴─────┐
   │          │
 Yes         No
   │          │
   ▼          ▼
 Return   Cooldown Provider
 Response       │
                ▼
        Select Next Provider
                │
                ▼
         Retry Request

Routing lifecycle

---

Key Features

OpenAI Compatible

Drop-in replacement for OpenAI APIs.

Simply change your base URL.

Compatible with:

• OpenAI SDK
• LiteLLM
• LangChain
• Direct HTTP clients

---

Multi-User Platform

Each user receives:

• Secure authentication
• Isolated provider credentials
• Personal OpenBalancer API key
• Independent routing decisions

Users never share provider credentials.

---

Intelligent Routing

Built-in routing policies:

• balanced
• fastest
• stable
• fallback
• cheapest

Routing decisions consider:

• Provider health
• Latency
• Quotas
• Failures
• Cost
• Cooldown state

---

Automatic Failover

If a provider becomes unavailable or reaches quota limits, OpenBalancer automatically retries using alternative providers whenever possible.

---

Supported Providers

Cloud

• Groq
• OpenRouter
• Gemini
• Cerebras
• Hugging Face
• TogetherAI (planned)
• Fireworks (planned)
• DeepInfra (planned)

Self Hosted

• Ollama (planned)
• vLLM (planned)
• Text Generation Inference (planned)
• llama.cpp (planned)

---

Dashboard Preview

Provider Management

Manage provider API keys directly from the dashboard.

• Groq
• OpenRouter
• Gemini
• Cerebras
• Hugging Face

---

Available Models

Browse models aggregated from connected providers.

---

SDK Quickstarts

Generate ready-to-use examples for:

• cURL
• Python Requests
• OpenAI SDK
• TypeScript

---

Quick Start

Clone the repository

git clone https://github.com/DevBhuyan/OpenBalancer.git

cd OpenBalancer

Install dependencies

python -m pip install -e ".[dev]"

Run the server

python -m openbalancer

---

Configure Provider Keys

OpenBalancer supports environment variables:

GROQ_API_KEY
OPENROUTER_API_KEY
CEREBRAS_API_KEY
GEMINI_API_KEY
HF_API_KEY

or provider credentials through the hosted dashboard.

---

Send a Request

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model":"auto",
  "messages":[
    {
      "role":"user",
      "content":"Hello!"
    }
  ]
}'

---

Routing Policies

Automatic model selection:

• auto
• auto:small
• auto:large

Available routing modes:

Policy	Description
fallback	Provider priority order
fastest	Lowest observed latency
stable	Lowest failure rate
balanced	Latency + failures + cost
cheapest	Lowest configured cost

---

Benchmarks

Benchmark Graph

Example benchmark scenarios:

• Capacity aggregation
• Provider failover
• Routing policy comparison
• Concurrent load testing
• Multi-user isolation

Run the benchmark suite:

python clients/load_test.py

---

API

Chat Completions

POST /v1/chat/completions

Models

GET /v1/models

Health

GET /health

---

Technology Stack

Backend

• Python
• FastAPI
• AsyncIO

Frontend

• HTML
• CSS
• JavaScript
• Jinja2 Templates

Storage

• SQLite
• PostgreSQL (planned)

Deployment

• Uvicorn
• Docker (planned)
• Kubernetes (planned)

---

Roadmap

• OpenAI-compatible API
• Intelligent routing
• Automatic failover
• Multi-user authentication
• Provider credential management
• Web dashboard
• Hosted deployment
• PostgreSQL migration
• Usage analytics
• Dynamic provider scoring
• Team workspaces
• Cost optimization
• Ollama / vLLM integration
• Prometheus metrics
• Kubernetes deployment

---

Contributing

Contributions, feature requests, and bug reports are welcome.

Feel free to open an issue or submit a pull request.

---

License

Apache 2.0 License.