LLM Router v1

LLM Router is a local OpenAI-compatible gateway for routing chat requests across multiple LLM providers. Apps call one endpoint with a Router API key, choose a virtual model such as smart-free, and the router executes that route through ordered provider/model steps with automatic fallback.

The project is designed to make apps provider-independent while making it easy to prefer free or low-cost models first, then fall back when a provider is rate limited, out of credits, offline, or timing out.

What It Does

• Exposes OpenAI-compatible GET /v1/models and POST /v1/chat/completions
• Lets Costbrain/Casdoor users log in and manage their own routes and provider credentials
• Stores provider API keys encrypted at rest
• Issues hashed Router API keys for client applications
• Treats routes as virtual models, so clients call model: "smart-free" instead of provider model IDs
• Supports ordered fallback across provider/model steps
• Tracks usage, token counts, status, provider/model used, and errors
• Includes a React dashboard for providers, routes, model selection, testing, usage, and API keys
• Runs locally without Docker using a double-clickable macOS launcher

Stack

• Backend: Node.js, Express, TypeScript
• Frontend: React, Vite, TypeScript
• Database: MySQL
• Auth: Costbrain/Casdoor SSO for dashboard login, local JWT sessions after SSO, Router API keys for client apps
• Secrets: AES-256-GCM encryption for provider keys
• UI ports: dashboard 47283, backend 47284

Current Dashboard

Overview

Shows request totals, token totals, success rate, estimated cost, active routes, and provider health signals.

Providers

Stores credentials for OpenAI-compatible provider endpoints. Known providers use fixed provider names so the app can attach provider-specific metadata, while custom providers can define their own name and endpoint.

You can also set a nickname for duplicate keys, for example:

• OpenRouter - main
• OpenRouter - backup
• Groq - free tier

Provider keys are encrypted before storage and are never exposed back to the browser.

Routes

Routes are reusable virtual models. A client app only sees the route name, while the router internally tries ordered provider/model steps.

Example route:

smart-free
1. OpenRouter / qwen/qwen3:free
2. OpenRouter / deepseek/deepseek-chat-v3.1:free
3. Groq / llama-3.1
4. OpenRouter / openai/gpt-4.1-mini

The route page supports:

• Route cards with total model count
• Full-width route detail view
• Expandable connection commands
• Linked provider selection
• Pop-out model picker
• Searchable model list
• Free / paid filtering
• Model metadata columns for provider, inputs, input price, output price, context, and max output
• Drag-and-drop route step ordering
• Route deletion

Model Picker

The model picker loads models from the selected provider's /models endpoint and normalizes OpenRouter-style metadata when available.

For OpenRouter, model calls are cached in memory for 30 minutes. Other providers can be enriched from the cached OpenRouter catalog when model IDs or names match. The picker also merges known free model metadata from frontend/free-llm-providers.json for providers where free-model data is hard to get directly.

Non-chat generation models such as image, video, audio, TTS, Whisper, Kling, Veo, Flux, Imagen, Stable Diffusion, and similar output-only models are filtered out of route model selection.

Playground

The Playground lets you test any route directly from the dashboard. It uses the same route execution engine as /v1/chat/completions, so it exercises real provider keys, fallback behavior, usage logging, and response normalization.

Controls include:

• Route selector
• Optional system prompt
• User message
• Temperature
• Max tokens
• Response output
• Provider/model used
• Token usage
• Expandable raw JSON response

API Keys

Generates Router API keys such as:

router_sk_xxxxxxxxx

Client applications use these keys instead of provider credentials. Keys can be revoked or deleted.

OpenAI-Compatible API

List Routes As Models

GET /v1/models returns the authenticated user's active routes in OpenAI model-list format.

curl http://localhost:47284/v1/models \
  -H "Authorization: Bearer router_sk_xxxxxxxxx"

Example response:

{
  "object": "list",
  "data": [
    {
      "id": "smart-free",
      "object": "model",
      "created": 1730000000,
      "owned_by": "llm-router"
    }
  ]
}

Chat Completions

curl http://localhost:47284/v1/chat/completions \
  -H "Authorization: Bearer router_sk_xxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smart-free",
    "messages": [
      { "role": "user", "content": "Hello from LLM Router" }
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'

JavaScript Client

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "router_sk_xxxxxxxxx",
  baseURL: "http://localhost:47284/v1"
});

const response = await client.chat.completions.create({
  model: "smart-free",
  messages: [
    { role: "user", content: "Write a short launch post for LLM Router." }
  ]
});

console.log(response.choices[0]?.message?.content);

Routing Behavior

The V1 policy engine is rule-based. It does not use AI to choose routes.

Flow:

Authenticate Router API key
Load route by requested model name
Load ordered route steps
Skip temporarily disabled provider/model pairs
Try the first enabled model
Normalize successful response to OpenAI format
On retryable error, classify failure and try the next step
Log usage and status
Return success or final router error

Retry/fallback conditions include:

• HTTP 429
• Quota exceeded
• Credits exhausted
• Timeout
• Provider offline
• HTTP 500, 502, 503, 504

Non-retryable conditions include:

• Invalid provider API key
• Invalid request
• Unsupported parameters
• User cancellation

Quick Start: Local Mac, No Docker

Install and start MySQL:

brew install mysql
brew services start mysql

Create the local database and user once:

mysql -u root < scripts/setup-local-mysql.sql

If your local MySQL root user uses a password:

mysql -u root -p < scripts/setup-local-mysql.sql

Then double-click:

Run LLM Router.command

The launcher:

• Creates backend/.env on first run
• Installs npm packages if needed
• Applies database schema and migrations
• Starts backend and frontend in one Terminal window

URLs:

• Dashboard: http://localhost:47283
• Backend: http://localhost:47284

Manual Development

Install dependencies:

npm install

Create backend/.env:

cp backend/.env.example backend/.env

Default local environment:

PORT=47284
DATABASE_URL=mysql://router:router@127.0.0.1:3306/llm_router
JWT_SECRET=replace-with-a-long-random-secret
ENCRYPTION_KEY=replace-with-32-byte-base64-key
APP_ORIGIN=http://localhost:47283
REQUEST_TIMEOUT_MS=60000
CASDOOR_SERVER_URL=https://login.costbrain.com
CASDOOR_CLIENT_ID=2784b7f00ced1c8886c0
CASDOOR_CLIENT_SECRET=replace-with-casdoor-client-secret
CASDOOR_ORGANIZATION_NAME=costbrain
CASDOOR_APP_NAME=Costbrain_MCP
CASDOOR_REDIRECT_PATH=/callback

Start both apps:

npm run dev:local

Run checks:

npm run check
npm run build

Docker Compose

Docker support is still present, but the current local workflow is optimized around Homebrew MySQL and the macOS launcher.

cp backend/.env.example backend/.env
npm install
npm run db:up
npm run dev

Apply schema:

mysql -h 127.0.0.1 -P 3306 -u router -prouter llm_router < backend/src/database/schema.sql

Provider Presets

Provider presets live in:

frontend/llm-providers.json

Add another OpenAI-compatible provider by appending:

{
  "id": "provider-id",
  "name": "Provider Name",
  "endpoint": "https://provider.example.com/v1",
  "icon": "hub",
  "hasFree": false,
  "noAuth": false,
  "modelCount": 0,
  "hint": "Optional setup note shown in the dashboard."
}

If the provider needs a user-entered endpoint, set:

{
  "id": "custom-provider",
  "name": "Custom Provider",
  "endpoint": "",
  "endpointEditable": true
}

Known free-model metadata lives in:

frontend/free-llm-providers.json

This file is used by the model picker to show additional free models for providers such as NVIDIA, Qwen, Z.AI, Mistral, Scaleway, SambaNova, and Groq when provider APIs do not expose enough pricing/model metadata.

Project Layout

backend/src
  api
    dashboard.ts
    v1/chat.ts
    v1/models.ts
  auth
  database
  encryption
  middleware
  policy
  providers
  router

frontend
  llm-providers.json
  free-llm-providers.json
  src
    main.tsx
    services/api.ts
    styles.css

scripts
  setup-local-mysql.sql

Security Notes

• Dashboard users authenticate through Costbrain/Casdoor SSO.
• The backend exchanges the Casdoor callback code server-side and then issues a local dashboard JWT.
• Router API keys are hashed before storage.
• Provider API keys are encrypted at rest.
• Provider credentials are only used server-side.
• Routes, provider keys, logs, and API keys are scoped per user.
• Use HTTPS and strong secrets before exposing the service outside local development.

V1 Scope

Implemented:

• Costbrain/Casdoor dashboard authentication
• Provider credential management
• Provider nicknames for duplicate keys
• Route editor
• Ordered route steps
• Drag-and-drop step ordering
• Model picker with metadata, search, free/paid filters, and non-chat filtering
• Basic rule-based fallback
• Usage logging
• Dashboard playground
• Router API key creation, revocation, and deletion
• OpenAI-compatible model listing and chat completions

Not included yet:

• Streaming responses
• Embeddings, images, audio, or responses API
• Semantic routing
• Latency or cost optimization
• Parallel racing
• Team accounts
• Billing
• Automatic benchmarking