The self-hosted LLM router for policy, cache affinity, and spend control in one Rust Binary.
Millwright is an open-source, self hosted LLM router that runs between your AI applications and the model providers you choose. It accepts OpenAI Chat Completions and Anthropic Messages, then routes each request to an OpenAI-compatible API, Anthropic, or Amazon Bedrock.
You define which models belong in the cheap, mid, and frontier roles. Millwright selects the lowest estimated-cost healthy route the resolved role permits and, when given a session ID, preserves provider/model affinity for prompt-cache reuse.
Millwright is a router, not an agent orchestrator. It does not spawn agents, schedule work, inspect prompts, or rewrite context.
LLM costs rise quickly when every request reaches the most expensive model and provider changes break prompt-cache reuse. Millwright makes the lower-cost path explicit:
- Control model spend — reserve frontier models for the work your policy says needs them.
- Protect prompt caches — role-scoped affinity keeps each session on a confirmed provider and model while its cache window is warm.
- Keep concurrent agents moving — one session can maintain independent
cheap,mid, andfrontierlanes without serializing requests. - Change providers without changing application code — route compatible traffic across providers behind one endpoint.
- Keep routing inspectable — response headers identify the chosen route; traces and ledger entries retain the evidence when persistence is available.
- Run the data plane yourself — provider credentials stay on your infrastructure, with no required hosted control plane.
Rust 1.97+ is required when building from source.
git clone https://github.com/Northwood-Systems/millwright.git
cd millwright
cargo install --path . --locked
millwright init
millwright servemillwright init walks through provider endpoints, model roles, pricing, and separate workload and operator keys. It stores references to provider-secret environment variables, never the provider credentials themselves.
Connect Claude Code, Codex, OpenCode, Pi, or another compatible client. Choose millwright-fast, millwright-code, or millwright-plan to request the cheap, mid, or frontier role.
The included Docker Compose setup starts Millwright with a mock provider and local demo key:
make compose-upFrom another terminal:
curl http://localhost:8080/v1/chat/completions \
-H 'Authorization: Bearer sk-millwright-local-demo' \
-H 'Content-Type: application/json' \
-d '{"model":"millwright-fast","messages":[{"role":"user","content":"Hello from Millwright"}]}'Architecture · Router API · Policy & Catalog · Deployment · Cost Analysis · Security
Classify requests with explicit task and risk headers, the client's model selection, or documented defaults. Millwright resolves a cheap, mid, or frontier role, then selects the lowest estimated-cost healthy model that policy allows. High-risk and planning requests always use the frontier role.
Every decision includes the chosen model, provider, route reason, and rejected alternatives. Message content is never inspected to make the decision.
Naive per-request routing can erase provider prompt-cache savings by switching models mid-session. Millwright keeps independent cheap, mid, and frontier affinity lanes under one opaque session ID, each with its own cache TTL.
Traffic resolved to different roles no longer fights over one warm route. Same-role calls converge on the last confirmed provider/model, while all requests remain concurrent—Millwright does not serialize agent traffic or orchestrate agent lifecycles.
Completed requests normally write usage evidence, cost provenance, provider attempts, and routing evidence to the ledger. SQLite is the local default; PostgreSQL is available for production deployments.
| Command | What it shows |
|---|---|
millwright spend |
Total spend, cache read rate, and per-team breakdown |
millwright models |
Model and provider mix |
millwright trace <id> |
One routing decision, including rejected alternatives |
millwright top |
Live spend and model activity in the terminal |
The same data is available through the authenticated /v1/millwright/* management APIs, with JSON output from spend, models, and trace for dashboards and automation.
millwright analyze reads a live Millwright ledger or local Millwright, OpenAI-compatible, or Anthropic JSONL. It produces a self-contained HTML or Markdown report plus schema-versioned JSON.
millwright analyze \
--url http://127.0.0.1:8080 \
--key "$MILLWRIGHT_OPERATOR_KEY" \
--since 30d \
--report analysis.html \
--json-out analysis.jsonReports separate measured cost evidence from modeled candidate economics. Add an explicit catalog and candidate allowlist to compare full-swap costs and a price-floor economic scenario. Candidate results are cost hypotheses—not quality results or production routing recommendations. See Cost Analysis.
OpenAI Chat Completions and Anthropic Messages come in; OpenAI-compatible APIs, native Anthropic, and Amazon Bedrock go out. Same-protocol routes preserve fields Millwright does not model. Cross-protocol routes translate the supported text-and-tools subset and reject unsupported shapes instead of silently dropping data.
Retryable transport failures, timeouts, HTTP 408/429, and HTTP 5xx responses can trigger one bounded failover attempt. Circuit breakers stop unhealthy routes from receiving every request.
Millwright ships as one Rust binary with Docker and Docker Compose support. Provider credentials remain in environment variables, while separate workload and operator keys control inference and management access. Structured logs omit API keys and prompt payloads; request bodies, concurrency, timeouts, and upstream buffers are bounded.
Need help deploying Millwright inside your infrastructure, integrating a provider, or designing a routing policy for your workload?
Bug fixes, features, tests, and documentation improvements are welcome. Read CONTRIBUTING.md before opening a pull request.
The Rust toolchain is pinned in rust-toolchain.toml.
git clone https://github.com/Northwood-Systems/millwright.git
cd millwright
make build
make testBefore opening a pull request, run:
make fmt
make lint
make test
make auditThe checks cover Rustfmt, Clippy with warnings denied, all Rust test targets, dependency advisories, approved licenses, and dependency sources. make audit requires cargo-deny. CI also builds the release binary and Docker image.
Apache-2.0