A learned quality predictor for LLM routing — and a measurement of what interpretability costs.
Predict whether a cheap 8B model will produce an acceptable answer before calling it, so a frontier model is only invoked when it's actually needed.
Status — work in progress. The pipeline is complete and runs end to end; the labeled inference run has not been executed, so there are no results in this repository yet.
results/summary.mdrenders every pending section asNOT YET PRODUCEDalongside the command that fills it — by design, so no placeholder number can survive into a draft. What is finished and testable today: 67-feature extraction, six router variants, the PGR/APGR/CPT metric suite, the failure taxonomy, and the generated write-up (python -m pytest tests/→ 15 passing;python scripts/smoke_test.py→ 10/10 checks).
Every query sent to a frontier model that an 8B model would have handled fine is money burned. Every query sent to the 8B model that it botches is quality lost. A router decides which is which, before generation, from the query alone.
Five systems already do this (RouteLLM, Martian, FrugalGPT, AutoMix, Hybrid LLM — reviewed in
research/prior_art.md). All five share two gaps:
- Every one of them is a black box. They route on embeddings or a fine-tuned transformer. Not one publishes an answer to what property of a query predicts that a small model will fail?
- Not one validates its labels. RouteLLM trains on Arena preferences, Hybrid LLM on BARTScore, FrugalGPT on task accuracy, AutoMix on self-verification. None reports an agreement statistic for the label it trains on — so every published number rests on an unmeasured error term.
RouteSmith does not claim a better cost-quality frontier than RouteLLM. Beating a matrix factorization router trained on 120k judge-labeled pairs with a logistic regression on ~10³ examples is not a realistic goal, and claiming it would be a red flag.
It claims a different deliverable: an explanation, validated.
| Prior work | RouteSmith | |
|---|---|---|
| Router input | Embeddings | 67 interpretable query features; embeddings kept as the control |
| Headline number | Cost saved | Share of the black-box router's performance an explainable router recovers |
| Error analysis | Error rate | Labeled taxonomy over every routing error, by cause |
| Label quality | Assumed | Measured: human-vs-judge κ, and judge-vs-ground-truth agreement |
| Error asymmetry | Symmetric | Failure rate reported and used to pick the operating point |
The experiment is not rigged: if interpretable features recover most of the embedding router's APGR, that's a strong, deployable, auditable result. If they recover much less, the finding is that routing signal is semantic and doesn't reduce to surface features — also a real result, also reported. The direction isn't known in advance.
There are none yet — see the status note above. When the run completes,
results/summary.md is the write-up and
results/figures/pareto_curve.png is the headline figure. Both are generated by
scripts/make_summary.py and analysis/plots.py
from pipeline artifacts, never hand-edited, so the prose and the numbers cannot disagree.
What blocks the run: API credentials for the two model providers, verification of the
small-model token prices in models/configs/pricing.json
(currently flagged verified: false, and they feed the cost axis directly), and 200
hand-labeled responses to clear the judge-validation gate.
python -m pip install -r requirements.txt
# Unit tests for the reported statistics — no API keys, seconds.
python -m pytest tests/ -v
# Prove the whole pipeline runs — no API keys, no cost, ~3 minutes.
python scripts/smoke_test.py
# Then set credentials and run for real.
export ANTHROPIC_API_KEY=... # large model + judge
export TOGETHER_API_KEY=... # small model (or GROQ_API_KEY, FIREWORKS_API_KEY, ...)
bash scripts/run_full_pipeline.sh --pilot # ~90 queries, cents, proves the wiring
bash scripts/run_full_pipeline.sh # the real runscripts/run_full_pipeline.sh --offline re-derives every published number from cached data
without calling anything — that's what makes the results reproducible by someone who has the
data directory but not the keys.
query ──> 67 interpretable features ──> LR / GBT ──> P(small model is acceptable)
(~1 ms, no model call) │
threshold ──┴──> small or large
Phase 0 — prior art. Five systems reviewed, with their metrics adopted (PGR / APGR / CPT
from RouteLLM) and their gaps identified. research/prior_art.md.
Phase 1 — data. Three benchmarks, chosen so the headline doesn't rest on an LLM's opinion:
| Benchmark | Task | Label | Why |
|---|---|---|---|
| MMLU | Knowledge MCQ | Exact match | Ground truth, no judge |
| MBPP-sanitized | Code generation | Execution against the benchmark's asserts | The strongest label in the project |
| Arena-Hard | Open-ended | LLM judge (validated) | Real traffic, selected to separate strong from weak models |
Both models answer every query; every call is priced from real token counts into
data/cost_log.jsonl.
Phase 1.3 — labeling, three tiers. Objective grading wins over everything. ≥200 hand-labeled responses validate the LLM judge (pre-registered bar: κ ≥ 0.6, set before seeing data). The judge also scores the objective slice, which buys a stronger check for free: agreement against ground truth rather than against another opinion. Both the human and the judge are blind to the other model's response and to each other's scores.
Phase 2 — features. 67 features across four hypotheses about why an 8B model fails —
complexity, knowledge demand, format constraints, linguistic ambiguity. Word rarity uses GPT-2
subword fertility (common words are one token, rare ones fragment), so no frequency corpus is
needed. Extraction is single-digit milliseconds against a 100 ms budget;
tests/test_metrics.py fails the build if p99 exceeds 25 ms, which is what caught the
per-word tokenizer call that originally cost ~160 ms/query.
Phase 3 — routers. Six variants (LR/GBT × interpretable/embedding/combined), 70/15/15 split stratified jointly on benchmark and label, 5-fold CV for hyperparameters, test set touched once. Three reference curves bound the plot: oracle (ceiling), random (floor), and length-only — routing by token count alone, the baseline that decides whether feature engineering earned its place.
Phase 4 — failure analysis. Every routing error categorized. False positives (quality lost) get a cause; false negatives (money wasted) get the feature that made the router flinch. Group ablation measures each feature family's worth in APGR, not AUC.
PGR can exceed 1.0, and that's not a bug. The frontier model isn't perfect. On queries where the small model succeeds and the large one fails, routing beats always-large — our oracle reaches PGR ≈ 1.11 (104% quality retention). This is the mechanism behind Martian's "beats GPT-4 by routing" claim. It isn't clamped; clamping would hide the most interesting region of the plot.
The failure rate is a separate number from PGR. PGR treats both error directions symmetrically; routing a query to the small model and getting a bad answer is not symmetric with wasting money. The recommended operating point is chosen under an explicit quality floor, stated in advance — not by maximizing a blended score after seeing the data.
Quality metrics use binary acceptability, not the 1–5 scale. The dataset mixes objective outcomes with judged scores; averaging a projected 5 against a judged 4 would produce a number with no defined units. The rubric scale is reported separately as a robustness check.
temperature=0 isn't available on the large model. Current Claude models reject non-default
sampling parameters. Raw responses are cached so every number is reproducible from stored
artifacts, but a fresh generation run wouldn't be bit-identical. See llm/client.py.
routesmith/
├── research/prior_art.md Phase 0 — five systems, and where RouteSmith fits
├── data/
│ ├── loaders/ MMLU, MBPP, Arena-Hard -> one schema
│ ├── schema.py RoutingExample; JSONL round-trip at every stage boundary
│ ├── responses/ raw model responses
│ ├── human_labels.jsonl Tier 2
│ └── cost_log.jsonl every API call, priced
├── llm/ provider clients (Anthropic SDK; OpenAI protocol for the rest)
├── features/
│ ├── extract.py 67 interpretable features
│ └── embed.py the control condition
├── models/
│ ├── train.py six routers + three references
│ └── configs/ models and pricing as data, not code
├── evaluation/
│ ├── grade_objective.py exact match + sandboxed execution
│ ├── judge.py blind LLM judge, structured output
│ ├── judge_validation.py κ, ρ, bootstrap CIs (hand-rolled; see tests/)
│ ├── label.py tier precedence + the Phase 1.3 checkpoint
│ ├── system_metrics.py PGR / APGR / CPT / Pareto / failure rate
│ └── cost.py the ledger
├── analysis/
│ ├── failure_analysis.py -> failure_taxonomy.md
│ ├── feature_importance.py coefficients, SHAP, ablation
│ ├── plots.py all figures
│ └── *.ipynb generated by scripts/make_notebooks.py
├── results/ summary.md, figures/ (both generated)
├── tests/test_metrics.py κ/ρ vs sklearn+scipy, PGR/APGR, latency budget
└── scripts/
├── build_dataset.py run_inference.py run_labeling.py
├── human_label.py smoke_test.py make_summary.py
└── run_full_pipeline.sh
Two directories deviate from the layout in the project spec, both deliberately: llm/ exists
because the inference runner and the judge need identical latency and token accounting, and
duplicating it would guarantee drift; analysis/*.py exists because the notebooks should call
tested modules rather than reimplement analysis in cells, so every figure is reproducible by a
script.
- No infrastructure theater. No Docker, no FastAPI, no dashboard. Scripts, notebooks, markdown.
- Every number is reproducible. If it's in the summary, a named script produced it — which is why the summary is generated rather than written.
- Costs tracked religiously. Every call logged with real token counts; unverified prices are flagged, not silently trusted.
- Interpretability over accuracy. A logistic regression 2% behind an opaque model is the better result here. The goal is explaining why small models fail, not only predicting it.
- Limitations stated up front.
results/summary.md§8 is written before anyone asks. - Prior-art awareness. Every design decision references whether prior work made the same choice, and why we diverged.