JAiRouter

Production-Ready AI Model Gateway

OpenAI-compatible API for unified routing, load balancing & failover
across Ollama, vLLM, GPUStack, Xinference, Claude, Gemini, and more

中文 • Docs • English Docs • Discussions

# Start with Docker (no configuration needed)
docker run -d --name jairouter -p 8080:8080 sodlinken/jairouter:latest

# Open Web Console: http://localhost:8080/admin
# Default: admin / ChangeMeOnFirstStartup123456

JAiRouter is a production-ready AI model gateway that provides a unified, OpenAI-compatible API for managing multiple LLM backends. It handles load balancing, rate limiting, circuit breaking, and failover — so you can focus on building applications, not managing infrastructure.

• 🔌 OpenAI-Compatible API — Drop-in replacement for OpenAI SDK, LangChain, LlamaIndex
• 🔧 Configurable Adapters — Add new AI providers (DeepSeek, Zhipu, etc.) via config or Web UI, no code needed
• ⚖️ Smart Load Balancing — Round-robin, weighted, least-connections, IP-hash, consistent-hash
• 🛡️ Rate Limiting — Token bucket, leaky bucket, sliding window algorithms
• 🔥 Circuit Breaker — Auto failover with configurable thresholds and recovery
• 🔐 Authentication — JWT + API Key dual authentication with audit logging
• 📊 Observability — Prometheus metrics, OpenTelemetry tracing, real-time dashboards
• 💾 Persistence — Redis / H2 / File storage for distributed deployment
• 🎛️ Web Console — Visual management, version control, configuration rollback

┌─────────────────────────────────────────────────────────────────┐
│                         Your Application                         │
│                 (OpenAI SDK / LangChain / LlamaIndex)           │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
                                  ▼ OpenAI-Compatible API
┌─────────────────────────────────────────────────────────────────┐
│                         JAiRouter Gateway                        │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐   │
│  │ Routing │ │ Balance │ │  Limit  │ │ Circuit │ │   Auth  │   │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘   │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              Observability & Persistence                 │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
       ┌──────────────┬───────────┼───────────┬──────────────┐
       ▼              ▼           ▼           ▼              ▼
   ┌───────┐     ┌───────┐   ┌───────┐   ┌──────────┐   ┌───────┐
   │Ollama │     │ vLLM  │   │GPUStack│  │Xinference│   │ OpenAI │
   └───────┘     └───────┘   └───────┘   └──────────┘   └───────┘

Add any OpenAI-compatible provider via configuration or Web UI:

📖 See Adapter Configuration Guide for details.

from openai import OpenAI

# Point to JAiRouter instead of OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # JAiRouter handles authentication
)

# Use any model from your configured backends
response = client.chat.completions.create(
    model="llama3.2",  # Routed to Ollama, vLLM, or GPUStack
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

# Via API - Add Ollama instance
curl -X POST http://localhost:8080/api/config/instance/add/chat \
  -H "Content-Type: application/json" \
  -H "Jairouter_token: your-jwt-token" \
  -d '{
    "name": "llama3.2",
    "baseUrl": "http://localhost:11434",
    "path": "/v1/chat/completions",
    "weight": 1
  }'

Nginx is a general-purpose web server. JAiRouter is purpose-built for AI/LLM workloads with OpenAI-compatible routing, circuit breaking, and model-aware load balancing.

One-API focuses on API key management and billing. JAiRouter focuses on local model gateway with advanced resilience patterns and observability.

LangChain is an application framework. JAiRouter is an infrastructure layer that works beneath LangChain to provide routing, failover, and monitoring.

Performance overhead compared to direct backend access:

Benchmarks: Ubuntu 22.04, 16 cores, 32GB RAM, Ollama 0.1.27

• Core gateway functionality
• Multiple backend adapters (Ollama, vLLM, GPUStack, Xinference, LocalAI)
• Load balancing with multiple strategies
• Rate limiting algorithms
• Circuit breaker with auto recovery
• Web management console
• JWT + API Key authentication
• OpenTelemetry distributed tracing
• Configuration version control
• Docker image optimization (Alpine/Distroless)
• API Key quota management
• Call history persistence
• RBAC role-based access control
• Anthropic Claude adapter
• Google Gemini adapter

Current Release: v2.8.3 | LTS Release: v2.6.11 (maintained until 2028-05)

We welcome contributions! See Contributing Guide.

git clone https://github.com/Lincoln-cn/JAiRouter.git
cd JAiRouter/modelrouter
mvn clean package -DskipTests
java -jar target/modelrouter.jar

• Documentation: https://jairouter.com
• Issues: GitHub Issues
• Discussions: GitHub Discussions

JAiRouter is released under the Apache 2.0 License.

Star ⭐ this repo if you find it useful!

Made with ❤️ by the JAiRouter Team