Model Router

Type a prompt → the app classifies what kind of task it is → routes it to the model that performs best on benchmarks for that task (optionally adjusted for cost) → streams the answer from that model using your own API keys.

How it works

1. Classify — app/router.py scores the prompt against keyword/pattern signals for 8 task categories (coding, math, reasoning, creative writing, summarization, long context, vision, simple/fast). Speculative and hypothetical questions ("who will win…", "what if…") count as reasoning.

2. Select — benchmarks.json holds a per-category score for each model (with the benchmarks the score is based on, e.g. SWE-bench Verified for coding, AIME for math) plus API pricing. The winner is the highest cost-adjusted score at your chosen budget level:

   • best — pure capability, cost ignored
   • premium — frontier quality, but big price gaps flip near-peers (max 10-point penalty)
   • balanced — cost weighs roughly as much as the typical capability spread (max 35 points)
   • economy — cheapest capable model wins almost always (max 60 points)

   The penalty scales with each model's blended price (3:1 input:output, log-normalized since prices span ~$0.25–$50/MTok). Weights live in BUDGET_WEIGHTS in app/router.py.

3. Execute (optional, BYOK) — the prompt is sent to the chosen model and the response is streamed back token by token. If the benchmark winner belongs to a provider you have no key for, the router falls back to the best-scoring model among your configured providers and tells you it did.

BYOK — bring your own keys

This is not a hosted-credits app. Visitors paste their own API keys (Anthropic / OpenAI / Google, plus optional Tavily / Brave for search) into the 🔑 panel:

• Keys are saved in the visitor's browser only (localStorage).
• They travel with each request over HTTPS and are used server-side for that request only — never stored, never logged, never read from server env vars.
• Routing decisions need no keys at all.

Web search

Check the web-search box (or pass "search": true) and the app fetches live results before executing, then hands them to the routed model to reason over. Backends are tried in order:

1. Tavily (user's tavily key — LLM-oriented results)
2. Brave Search (user's brave key)
3. DuckDuckGo — keyless, works with zero setup. Note: on shared serverless egress IPs it may get rate-limited, so a Tavily or Brave key is recommended for deployed instances.

Findings (with source URLs) are appended to the prompt inside a <web_search_findings> block; the pipeline used is shown above the answer.

Run locally

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload

Open http://127.0.0.1:8000 and add keys in the 🔑 panel.

Deploy to Vercel

The repo is Vercel-ready:

• api/index.py exposes the FastAPI app as a serverless function; vercel.json rewrites all routes to it and sets maxDuration: 300 so long generations complete (streaming keeps the connection alive within that window).
• No server-side secrets are required — the app is fully BYOK.

Steps:

git init && git add -A && git commit -m "model router"
# push to GitHub, then import the repo at vercel.com/new
# (framework preset: Other — Vercel auto-detects the Python function)

Notes for the deployed instance:

• Long max_tokens runs on frontier models can still approach the 300s cap — responses stream, so partial output is visible immediately.
• DuckDuckGo may be flaky from shared serverless IPs (see Web search above).
• Keys are the visitor's own, so there is no server credit to protect; if you ever add server-side keys, put an access gate and rate limiting in front.

API

• POST /api/route — {"prompt", "budget", "only_available", "keys": {...}} → routing decision (category, chosen model, full cost-adjusted ranking, missing-key fallback).
• POST /api/execute — same body plus "search" → SSE stream: a meta event (decision + pipeline), then delta events with response text, then done or error.
• GET /api/benchmarks — the current benchmark registry.

Customizing

• Add a model: append an entry to benchmarks.json with per-category scores and cost; add a streaming adapter in app/providers.py if it's a new provider.
• Tune routing: edit CATEGORY_SIGNALS / BUDGET_WEIGHTS in app/router.py.
• Refresh scores: benchmarks.json is data-only — update it from current leaderboards and the router picks up the changes on the next request.