🚀 Flow-LLM-Router: The Ultimate "Token Saver"

• ⚡️ The Most Cost-Effective LLM Platformn at FlowAPI Platform.

• 🐈 Flow-LLM-Router Your Local AI Control Plane to Save Token.

• 📊 Automated Token Analytics Dashboard

• 🧠 Smart API Routing (Rules & Classifiers)

• 🧰 Dynamic Skill Loading (Eliminate the "Token Tax")

• 🔒 Secure, Local API Key Management

• The Problem
• Why Flow LLM Router
• Who It Is For
• Use Cases
• Why Pair It With FlowAPI
• Key Capabilities
• Quick Start
• Documentation
• FAQ

• OpenAI-compatible by default: point existing SDKs to http://host:7798/v1 and keep most client code unchanged.
• Local-first observability: request logs, token usage, latency, and routing metadata stay in local SQLite.
• Encrypted provider vault: provider API keys are stored encrypted at rest and unlocked only when needed.
• Multi-provider routing: route across OpenAI, Anthropic, Gemini, DeepSeek, Qwen, Groq, custom OpenAI-compatible endpoints, and more.
• Operator-friendly dashboard: manage providers, models, caller tokens, logs, analytics, router settings, and integration snippets from one UI.
• Optional smart router: use fast rule-based complexity scoring or RouteLLM-powered classifier routing.

• teams building AI agents, copilots, or workflow automation on top of OpenAI-style SDKs
• developers who want local logs and routing visibility without adopting a hosted observability layer
• operators who need one stable endpoint in front of multiple model providers
• builders who want to combine lower-cost upstream pricing with local request optimization

• AI agents and copilots: centralize routing and provider auth behind one OpenAI-compatible endpoint
• Multi-model workflows: send easy tasks to cheap models and reserve premium models for high-value reasoning
• Private internal tooling: keep prompts, logs, and credentials inside your own environment
• Cost debugging: identify which models, prompts, and traffic patterns are silently increasing your bill
• Gateway standardization: give multiple applications one stable base URL even when upstream providers differ

Your App / Agent / SDK
        |
        | OpenAI-compatible requests
        v
  Flow LLM Router Proxy  ------------------------------+
        |                                       |
        | auth + routing + logging              |
        v                                       |
  LiteLLM forwarding layer                      |
        |                                       |
        +--> Provider vault (encrypted keys)    |
        +--> Smart router service               |
        +--> SQLite logs + model catalog        |
        +--> Static dashboard UI                |
        |
        v
OpenAI / Anthropic / Gemini / DeepSeek / Qwen / custom OpenAI-compatible backends

Flow LLM Router and FlowAPI solve different layers of the cost stack:

If you are serious about cost control, you usually want both.

Flow LLM Router is not magic. It improves cost structure through a few concrete levers:

Flow LLM Router exposes an OpenAI-style API surface so existing clients can usually switch by changing only the base URL and token source.

Supported endpoints today:

• POST /v1/chat/completions
• POST /v1/embeddings
• GET /v1/models

Streaming chat completions are forwarded as SSE.

Every proxied request can be logged with:

• requested model and routed model
• provider
• prompt, response, and error status
• token usage
• latency
• smart-router score and tier
• session metadata

The dashboard surfaces this through overview metrics, timelines, provider/model breakdowns, and per-request log detail views.

Flow LLM Router separates caller access from provider credentials:

• Caller tokens control who is allowed to use the proxy.
• Provider keys are stored encrypted in SQLite.
• Master-password unlock protects the provider vault.
• IP allowlisting limits where requests may come from.
• Log redaction helps prevent accidental credential leakage in persisted logs.

If no caller tokens exist yet, proxy access remains open for easier local setup. Once tokens are created, valid caller tokens become mandatory.

Flow LLM Router supports three routing modes:

• off: pass through the requested model unchanged
• complexity: local rule-based routing using a 7-dimension prompt complexity scorer
• classifier: RouteLLM-based routing, mapped back into Flow LLM Router's four-tier model layout

Both routing strategies use the same tier mapping:

• SIMPLE
• MEDIUM
• COMPLEX
• REASONING

If RouteLLM is unavailable or fails at runtime, Flow LLM Router falls back to rule-based routing instead of breaking requests.

Flow LLM Router includes a built-in local dashboard so you can inspect:

• which models are used most often
• which providers consume the most tokens
• which requests are slow, error-prone, or over-routed
• how routing tiers are distributed over time

Cost optimization gets much easier once the waste is visible.

Flow LLM Router includes optional skills-related configuration and package extras for teams exploring retrieval- and tool-oriented prompt optimization.

In the current repository, this is best understood as a foundation for prompt-efficiency work rather than a fully productized dynamic skill-routing system.

pip install flow-llm-router

# RouteLLM-based classifier routing
pip install 'flow-llm-router[classifier]'

# ChromaDB-backed skills support
pip install 'flow-llm-router[skills]'

flow-router start

Default endpoints:

• Dashboard: http://127.0.0.1:7798
• Proxy base URL: http://127.0.0.1:7798/v1
• OpenAPI docs: http://127.0.0.1:7798/docs

To load a custom config file:

export FLOWGATE_CONFIG=/path/to/flow_llm_router.yaml
flow-router start

Go to http://127.0.0.1:7798 and:

• set up or unlock the vault
• add one or more provider keys
• optionally sync provider models
• optionally create caller tokens
• optionally configure the smart router

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:7798/v1",
    api_key="fgt_your_caller_token_or_dummy",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello from Flow LLM Router"}],
)

print(response.choices[0].message.content)

If you have not created any caller tokens yet, Flow LLM Router accepts requests without token enforcement. In production, create caller tokens and restrict access explicitly.

Copy the example file and adjust it for your environment:

cp flow_llm_router.yaml.example flow_llm_router.yaml

Main configuration areas:

• server: bind host and port
• smart_router: routing strategy and tier mappings
• skills: optional retrieval support
• database: SQLite database path
• logging: prompt/response logging and secret redaction
• security: vault, auth token TTL, persisted master key path, and IP allowlisting

Example smart-router configuration:

smart_router:
  enabled: true
  strategy: complexity   # complexity | classifier | off
  tiers:
    SIMPLE: gpt-4o-mini
    MEDIUM: gpt-4o
    COMPLEX: claude-sonnet
    REASONING: o1-preview

Environment references such as ${OPENAI_API_KEY} are supported in YAML values.

Use flow-router --help for all flags and options.

git clone <your-repo-url>
cd flow-llm-router
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,classifier]'
pytest -q

Frontend workflow:

cd frontend
npm ci
npm run build

To build the static dashboard and copy it into the package:

bash scripts/build_frontend.sh

No. This repository is the local-first gateway layer you run yourself.

No. Flow LLM Router uses LiteLLM as the forwarding layer and adds local routing, security, and observability on top.

Usually no. In most cases you only change the base URL and, if enabled, the caller token.

Yes. Flow LLM Router works independently with direct provider APIs and custom OpenAI-compatible endpoints. FlowAPI is an optional upstream pairing for better token pricing.

Not as a finished production feature in the current repository. The project includes skills-related configuration and extension hooks, but the README positions this today as an optimization direction and foundation rather than a fully shipped headline workflow.

Request metadata and catalog data live in local SQLite. Provider API keys are stored encrypted and unlocked only when needed.

Flow LLM Router is currently alpha and focused on shipping a tight local gateway experience for AI developers and small teams.

Current strengths:

• OpenAI-compatible local proxying
• encrypted provider credential handling
• integrated analytics and logs
• configurable smart routing
• model catalog sync
• local dashboard operations

Current gaps:

• no formal multi-node deployment story yet
• no official container or Helm distribution in this repository yet
• no guaranteed backward-compatibility promise across early releases

• OpenAI-compatible local proxy
• Encrypted provider vault
• Model catalog sync
• Smart router with rule-based and classifier modes
• Local analytics dashboard
• Richer cost attribution across agents and workflows
• Better router evaluation and decision explainability
• Production-ready packaging and deployment examples
• Stronger admin auth and multi-user operations
• More polished benchmarking, demos, and screenshots

The current codebase suggests a clear next path for the project:

• broaden provider ergonomics for custom OpenAI-compatible gateways
• deepen router evaluation and cost/quality observability
• improve deployment and packaging workflows
• harden authentication and admin-session handling
• expand documentation and operator guidance

Pull requests are welcome.

Good contribution areas:

• provider integrations and OpenAI-compatible endpoint handling
• routing strategy improvements and classifier evaluation
• analytics and dashboard polish
• packaging, deployment, and operator workflows
• documentation, examples, and benchmark material

If you are extending behavior, keep the docs aligned with the implementation and prefer changes that remain inspectable and local-first.

MIT