v0.8.0b1 pre-release

ModelSpend 0.8.0b1 (beta)

See what your LLM usage would have cost on other models, in the first minute, from the export your provider already gave you. Then check that your provider's numbers can be trusted.

ModelSpend is a small open-source tool (Apache-2.0) that sits between your code and the LLM providers you already use (OpenAI, Anthropic, Google, xAI, DeepSeek and anything OpenAI-compatible, with your own keys). It shows where your tokens and money go per project and agent, enforces budgets before the spend, and can route each task to the lowest-cost model expected to do the job. It runs locally, stores no prompts, and is built for one developer or a small team (1 to 25 people).

Try it in two minutes

git pull && pip install -e .          # or: pip install -e . in a fresh clone
# Save your provider's usage export (OpenAI: the export on its Usage page) as usage.csv, then:
modelspend whatif --from-export usage.csv --export-provider openai

No traffic through ModelSpend needed, nothing stored. On a fresh install it prices your tokens on the four shipped Anthropic models and tells you which commands add prices for the rest (modelspend keys set openai, modelspend catalog sync, modelspend catalog price ...).

What's new

What-if price table (modelspend whatif, GET /v1/whatif, and a table on the dashboard's Savings tab)

• Replays the tokens in your own usage ledger against every priced model in your catalog and shows what the same traffic would have cost, cheapest first, next to what you actually spent. It uses the same arithmetic that priced the real calls, so it cannot disagree with your ledger about a model you really used.
• Beside each price: the shadow evidence (passed/scored) and the task types where the model clears the same evidence bar routing uses. A model with no evidence says unmeasured, never "safe". A total if moved only there column prices your traffic with only those task types moved, so the headline is never "everything on the cheapest model".
• Models with no known price are listed apart, never guessed. A model priced at $0 (local or free) is labelled as having no per-token price.
• --from-export does the same from a provider's own usage export (CSV, JSON or JSON lines; the files reconcile import reads). Totals per model per day, so long-context price tiers cannot be applied (it says so); cached tokens are split out once for providers that count them inside input; spend with no cost column is called an estimate from catalog prices, not an invoice.

A report you can hand to someone (modelspend share, a download button on the Savings tab)

• One static HTML file (or --format md to paste into an issue): spend, projected saving, the evidence per task type, spend by model and the what-if table, with the caveats printed on the page.
• Safe to share by construction: numbers, model names and task types only. No prompts, responses, keys or file paths. Project names only with --include-projects. The HTML has no scripts and no network access (a Content-Security-Policy forbids both), every value is escaped, and it follows light and dark mode and works at phone width.

Check a provider on your own key (modelspend verify)

• A few tiny requests per provider (it asks first and prints the most it can cost: a fraction of a cent) check what your cost numbers depend on: the key works, an unknown model fails with an error ModelSpend understands, a completion reports token usage, a stream reports matching usage, structured output parses where the catalog claims it, and with --cache that cached tokens are counted once.
• A FAIL means a number ModelSpend shows for that provider would be wrong. --report writes a Markdown report (ids, counts and error kinds only) you can paste into an issue. This exists because fakes cannot find the failures that matter most.

Prompt-caching advice (modelspend caching, a table on the Savings tab)

• Shows which long prompt starts (system prompt, few-shot examples, a document, plus tools and schema) repeat in your traffic, how many cached tokens the provider actually reported for them, how many it missed, what caching has saved you and what is still on the table. When a repeated prefix is not being cached it names the usual cause (a timestamp or id near the top of the prompt).
• Anthropic needs explicit cache markers, which ModelSpend does not send yet, so those rows are information only. Providers other than OpenAI, DeepSeek and Anthropic are reported as "unknown" rather than assumed; modelspend verify <provider> --cache tests them.
• This stores something new, so read this: per request, for each prompt start of at least ~1,000 tokens, a keyed hash (HMAC with a random key that lives only in your own database) and a token count. Never the text. Nothing is tokenized for short prompts, and a tracking failure cannot affect a request. Turn it off with "track_prefixes": false in settings.json; modelspend caching --forget erases it.

More providers

• DeepSeek is a built-in preset, and any OpenAI-compatible endpoint (Groq, Together, Mistral, a local Ollama or vLLM, ...) can be described in settings.json under custom_providers, validated strictly (https only except localhost, no credentials in the URL, no clash with a built-in name).
• modelspend catalog import-openrouter --only <ids> brings OpenRouter models into your catalog with USD prices, context and output limits taken from OpenRouter's own model list. Dry run by default; the response is treated as untrusted; models arrive with no tier, so routing will not pick them on a prior, but shadow evaluation can measure them. :batch variants (half price, apparently asynchronous) are skipped unless you pass --include-batch.
• modelspend catalog remove 'openrouter/*:batch' --yes takes back models you added (dry run first; shipped models are never touched; the catalog is re-validated and the old file kept as overrides.json.bak).

Smaller

• "No eligible model" now says what to run for the usual causes. The README opens with install-and-try. CI builds the wheel, checks it and runs the installed commands on every push. A PyPI publishing workflow exists but is not set up yet: install from GitHub for now.

Verified live (2026-10-19 to 2026-10-20)

• OpenAI gpt-4.1-nano, modelspend verify --cache: every check passed, twice. Including cached tokens: the same ~1,300-token prompt sent twice reported 183 uncached + 1,152 cached = 1,335, the same total as the first call, so cached tokens are counted once.
• whatif --from-export on a real OpenAI usage export (202 requests, 3 models, 4 days): the dated model names matched the catalog, and the model prices, recomputed by hand from the token counts, matched.
• whatif and share on a real ledger (92 calls). The shared file was scanned for keys, key-shaped strings and local paths: only the repository link matched.
• Local Ollama (llama3.2:1b): the adapter answers a direct and a streamed call with token usage, and an unknown model fails with a classified error.
• OpenRouter importer against the live model list: the response parsed as documented, slash ids came through intact, and the three OpenAI models' prices matched the maintainer's own catalog. It also showed the :batch twins above, which the fakes could not.

Not yet verified

• The caching advisor has not been run on real traffic. The arithmetic is tested on constructed timelines and the provider behaviours it assumes (OpenAI and DeepSeek cache automatically, Anthropic needs markers) come from their documentation. Only OpenAI's cached-token reporting has been confirmed live.
• DeepSeek, Gemini and xAI have not been run against the live APIs (modelspend verify exists to do exactly that; reports are welcome). A call routed through OpenRouter and a routed call to Ollama are also unconfirmed; so is whether OpenRouter's per-token prices include every fee you will pay.
• Savings are projections, not measurements, and "passed" means as good as the answer you used, not correct. A cheaper model that matches your current one on classification says nothing about other task types.
• A cost comparison against a provider's invoice (token counts were reconciled against OpenAI's export in earlier releases); the Docker sandbox; automatic price refresh against real pages; concurrent budgets across processes; Windows.
• One intermittent test failure was seen once in about twelve full test runs and has not been identified. Eleven consecutive runs passed afterwards. If CI shows an intermittent failure, the test name would help.
• Everything listed as not yet verified in 0.7.0b1 that is not mentioned above still applies.

Upgrade

git pull && pip install -e .

Prompt-prefix tracking is on by default (hashes and counts only; see above). To turn it off before your next request: add "track_prefixes": false to ~/.modelspend/settings.json.

Full changelog: v0.7.0b1...v0.8.0b1

v0.7.0b1 Pre-Release

ModelSpend 0.7.0b1 (beta)

Find out which cheaper model is safe to use for your work, from your own requests, and see what it would save. No eval cases to write.

ModelSpend is a small open-source tool (Apache-2.0) that sits between your code and the LLM providers you already use (OpenAI, Anthropic, Google, xAI, with your own keys). It shows where your tokens and money go per project and agent, enforces budgets before the spend, and can route each task to the lowest-cost model expected to do the job. It runs locally, stores no prompts, and is built for one developer or a small team (1 to 25 people).

Try it in two minutes

git pull && pip install -e .          # or: pip install -e . in a fresh clone
modelspend shadow set --rate 0.05 --budget 5    # sample 5% of requests, never spend more than $5 a month
modelspend shadow on
# ... use your tools as normal through ModelSpend (SDK, CLI, or the local OpenAI-compatible server) ...
modelspend savings

Want to see it work right now for a few cents? python examples/shadow_live_check.py sends 30 ticket classifications to your premium model, lets ModelSpend try the cheaper ones in the background, and prints what it measured.

What's new

Shadow evaluation (off by default; you opt in)

• A small sample of the requests you already send is also tried, in the background, on cheaper models, and each answer is compared with the one you actually used: your validators if the request has them, else an exact match (good for labels), else a judge model.
• Private by design: only a verdict, a fixed reason code, costs and token counts are stored. Never the prompt, either answer, or anything the judge said. A test scans every table in the database for a planted prompt and answer.
• Spend is capped: a monthly ceiling (default $5), checked against each call's worst case before it is made, and every shadow call is in your usage ledger under project=modelspend-shadow. Requests marked sensitive are never shadowed unless you allow it. Your own request is never delayed, and a failure inside shadowing cannot reach you.

Savings projection (modelspend savings, GET /v1/savings, and a Savings tab in the dashboard)

• Per task type: what you spend, the cheapest model that clears the evidence bar, how many comparisons it passed, its 95% lower bound, its measured cost relative to the model you use, and the projected saving.
• A model is only "ready" when the same bar routing uses (your own thresholds, else the profile's) is met by the 95% Wilson lower bound. A handful of lucky samples never qualifies; everything else says what is missing ("needs 6 more", "below the bar").
• It is a projection, not a measurement, and "passed" means as good as the answer you used, not correct. The report says so.

Let routing use it, safely

• modelspend shadow apply shows which task types would change, then (with --yes) lets routing treat shadow results as evidence only where you have no eval evidence of your own. modelspend shadow revert undoes it.
• No chaining: answers from a model that qualified by shadow evidence are never used as the reference for further comparisons, so "A is close to B, B is close to C" cannot quietly erode quality.

See why, not just what

• modelspend shadow status lists, per model, why each comparison ended as it did (same answer, passed your validators, judge said no, the call failed and with what kind of error, ...). Reason codes only, never text.
• A candidate cut off at your max_output_tokens is not counted for or against it and is shown separately. Reasoning models spend output tokens thinking, and found live: gpt-5-mini returned empty answers at a 64-token cap and passed 28 of 28 comparisons at 512. After changing prompts or limits, modelspend shadow reset forgets stale comparisons (your spend records are kept, so the monthly cap still holds).

Using it with a few people

• Run one server for the team (modelspend serve --host 0.0.0.0 with MODELSPEND_API_TOKEN set; a non-loopback bind is refused without a token). Everyone points an OpenAI-compatible client at it and tags requests with X-ModelSpend-Project / X-ModelSpend-Agent. Spend, budgets and shadow evidence are then shared, and evidence builds faster because it pools everyone's requests.
• Today that is one shared token with no per-person identity or roles: anyone with the token can change settings. Fine for a few people who trust each other; not a multi-tenant service.

Guide, privacy, cost and limits: docs/shadow.md.

Fixed

• modelspend prune kept shadow comparisons when there were no old requests; it now deletes them.

Verified live (OpenAI, 2026-10-17)

• Two runs of 30 ticket classifications pinned to gpt-4.1, with gpt-4.1-nano and gpt-5-mini as the cheaper candidates, for $0.0073 of shadow spend in total against a $0.50 cap.
• gpt-4.1-nano: 57 of 57 scored comparisons passed (3 unscored: the judge's reply could not be read, which is now parsed more tolerantly), 95% lower bound 0.94, about 95% cheaper on those requests, so the report marked classification ready.
• gpt-5-mini: cut off at a 64-token cap, then 28 of 28 at 512 (above).
• The dashboard's Savings tab was checked in a browser in light, dark and phone width; the settings form saves and the server rejects invalid values.

Not yet verified

• Thirty easy requests of one task type prove the mechanism, not a saving. Real, varied traffic is the real test: how often real judge models agree with validator and exact-match verdicts, and whether the projected saving matches what happens after you switch.
• Agreement with your current model is not correctness. Without validators a candidate can agree with a wrong answer, and a judge model has its own biases. Use validators where correctness matters; your own eval evidence always wins over shadow evidence.
• The shared-server setup for a few people is tested but has not been used by a real team yet.
• The monthly cap is checked per process (one server process is fine). Streamed requests are not shadowed.
• Everything listed as not yet verified in 0.6.0b1 and 0.6.0b2 still applies (Gemini and xAI live, cost reconciliation against a real costs export, the Docker sandbox, automatic price refresh, Windows).

Upgrade

git pull && pip install -e .

Shadow evaluation does nothing until you run modelspend shadow on.

Full changelog: v0.6.0b2...v0.7.0b1

v0.6.0b2 Pre-Release

ModelSpend 0.6.0b2 (beta)

A fix release on top of 0.6.0b1, found by running live calibrations against OpenAI and reading the failures. If you run eval calibrate or eval run with a reasoning model, upgrade: the earlier results can understate it.

Fixed

• Reasoning models were scored as wrong when they ran out of output tokens. Evals cap each answer at 400 tokens. On harder cases gpt-5-mini spent almost all of that thinking (4,032 of its 4,156 output tokens were reasoning) and returned no visible answer, so its 0/2 on analysis, rewriting and summarization measured the cap, not its quality. Saved as evidence, that would have unfairly kept it out of routing. Now eval calibrate and the two baseline arms of eval run retry a cut-off call once with 1,024 more output tokens and judge the retry. Both calls are recorded in the usage ledger, the retry is skipped if it would break --budget, and a call that is still cut off counts as a failure labelled truncated at the output limit. eval calibrate prints how many calls were retried; eval run reports output_limit_retries. The ModelSpend arm of eval run is unchanged: it is the product's real behaviour, which escalates on a cut-off answer. See docs/own-evals.md.
• A ledger reused for two calls double counted the first. run_single recorded the whole ledger on every call, so the retry above would have re-recorded the first call's attempts. Each call now records only its own attempts. (Found by the new tests, not in live use.)
• An ambiguous built-in calibration case. cal-extraction-2 asked for "every person named... in order of appearance" but expected only the three distinct people, so gpt-4.1 and gpt-5-mini answered [Maria Chen, Dr. Okafor, Luis Prado, Maria, Luis], a fair reading, and failed. It now asks for each distinct person once, using the fullest name given. The built-in dataset is renamed modelspend-calibration-v2 so evidence measured on the old wording is not mixed with the new; re-run calibration if you stored evidence from the built-in set.

Verified live (OpenAI, gpt-4.1, gpt-4.1-nano, gpt-5-mini, 2026-10-16)

• The retry works: across two runs, 12 and then 10 calls stopped at the output limit and were retried; 11 and 9 of them recovered and were scored on their answers. gpt-5-mini went from 0/2 to 2/2 on rewriting and summarization and from 0/2 or 1/2 to 2/2 on code.
• What is still cut off is reported honestly: gpt-5-mini on one open-ended analysis case stays truncated even with 1,424 tokens, on both runs. That is worth knowing before routing analysis to it.
• After the case fix, all three models pass the extraction case (2/2).
• A 48-call calibration costs about $0.02 without retries and about $0.04 with them, inside a --budget 0.10; the retries are visible as extra requests under project=modelspend-eval.

Not yet verified, and known limits

• Routed (non-eval) calls are not changed. If you set a small max_output_tokens and the router picks a reasoning model, the answer can come back empty. With validators it shows as a failed not_truncated check and may escalate (costing more); without validators the status is truncated. Give reasoning models room, or pin a non-reasoning model for short answers. Sizing the cap automatically changes budget arithmetic and is not done. See docs/limitations.md.
• The post-fix day has not been reconciled against OpenAI's own usage export yet. The ledger total matched the calibration's printed spend exactly; comparing it with OpenAI's numbers is the remaining step.
• Sixteen cases cannot rank models. The built-in set is a smoke test (almost everything passes); use your own cases, or the harder example set in examples/evals/support-tickets-hard, to compare.
• Everything listed as not yet verified in 0.6.0b1 still applies: cost reconciliation against a real costs export, whether OpenAI's input tokens include cached tokens, Anthropic/Google/xAI export formats, Gemini and xAI adapters, concurrent budget enforcement, the Docker sandbox, automatic price refresh, Windows.

Upgrade

git pull && pip install -e .

Full changelog: v0.6.0b1...v0.6.0b2

v0.6.0b1 Pre-release

ModelSpend 0.6.0b1 (beta)

ModelSpend shows where your LLM tokens and money go, per project and per agent, enforces budgets before the spend, and can route each task to the lowest-cost model expected to suit it. It uses your own API keys for OpenAI, Anthropic, Google and xAI, stores no prompts, runs locally, and is Apache-2.0.

This release is about checking ModelSpend's numbers against the provider's. The first real check found a gap in ModelSpend's own accounting, which is fixed here. It is still a beta; what was and was not tested live is listed below.

What's new

Reconcile with the provider's usage

• Download a usage export from the provider (CSV, JSON or JSON lines), then:

  modelspend reconcile import ~/Downloads/completions_usage.csv --provider openai
  modelspend reconcile run              # per model, over the days the export covers
  modelspend reconcile run --by-day     # one row per model per day (UTC)
  modelspend reconcile run --check      # exit status 1 if anything needs a look (cron/CI)

  Nothing is fetched and no extra key is needed. Only numbers are read from the file; no text from it is stored.
• ModelSpend recording more than the provider billed is always flagged (token estimates, a different key or project than the one exported, or provider data that has not caught up). The provider reporting more is normal, because the provider counts everything on the key; it is only flagged with --exclusive.
• Differences are judged within a tolerance (default 2%, with a small floor so rounding on tiny amounts is ignored). Cached tokens follow each provider's convention and can be overridden. Model names with a date suffix merge (gpt-4.1-nano-2025-04-14 is gpt-4.1-nano). Days the provider reports empty count as covered, so a ModelSpend call on such a day is flagged.
• A match does not prove either side right: the provider's invoice is authoritative. Guide and limits: docs/reconcile.md.

Fixed

• Live evals were missing from the usage ledger. The first reconciliation against OpenAI's export showed about 35,000 tokens (about 190 requests) on one day that were in OpenAI's numbers but not in modelspend report. They were eval run and eval calibrate calls: those commands ran without a usage store, so their real spend was invisible to reports, the Costs tab and budgets. Their calls are now recorded under project=modelspend-eval (agent=calibrate or eval-run), so they show up everywhere and can be isolated or excluded with --project. Stickiness is switched off for eval requests, so recording cannot change which model a case is routed to. Evals run before this release were not recorded and cannot be recovered, so a reconciliation over those days will show the provider above ModelSpend.
• First use from two processes on one database could crash after the provider call. Two processes both seeing "no salt yet" made the second fail with UNIQUE constraint failed: meta.k. It now keeps whichever salt won. This surfaced as a flaky concurrent-budget test (about 1 run in 6).
• Reconcile output: a ModelSpend row with zero tokens and nothing from the provider (a call that failed before billing) is no longer flagged, and the result line explains why provider-higher rows are not flagged.

Verified live

• Token counts match OpenAI's exactly on the days checked: on 2026-10-12 and 2026-10-14, ModelSpend's totals per model equalled OpenAI's usage export for gpt-4.1-nano (111 and 181 tokens) and gpt-5-mini (201 tokens), with no difference.
• The OpenAI completions usage CSV export imports correctly, including its blank rows for quiet days and its float-formatted numbers.
• The reconciliation is also what found the eval-ledger gap above.
• From earlier releases: routing, streaming, structured output (strict and non-strict) and failover and escalation on OpenAI; Anthropic end to end; per-project and per-agent reports, budgets, the Costs tab, aliases, conversation stickiness and your own evidence bars.

Not yet verified

• Cost reconciliation. OpenAI's usage export has no cost column, so only tokens were compared. Cost handling is tested but has not been checked against a real costs export.
• Whether OpenAI's input_tokens already includes cached tokens is assumed (the default for OpenAI). The sample export had no cached tokens.
• Anthropic, Google and xAI export formats. Their column names and JSON shapes are based on documentation, not real files. --map covers most naming differences.
• The fix for the eval gap itself: a day of live evals after the fix, followed by a reconciliation, has not been run yet; the fix is covered by tests.
• Fetching usage straight from the providers' admin usage APIs is not built; it would need an admin-level key.
• Gemini and xAI adapters (tested against simulated responses only), concurrent budget enforcement across processes and the dashboard's budget form, the Docker code sandbox, and automatic price refresh against the real pages.
• macOS and Linux with Python 3.10+ only; Windows is untested.

Upgrade

git pull && pip install -e .

Full changelog: v0.5.0b2...v0.6.0b1

v0.5.0b2 Pre-Release

ModelSpend 0.5.0b2 (beta)

A small fix release on top of 0.5.0b1. If you call the Python SDK's sync methods more than once, upgrade.

Fixed

• Event loop is closed on the second sync SDK call. ModelSpend.route(), preview() and stream() ran every call on a fresh event loop and closed it afterwards. The OpenAI, Anthropic and Google SDK clients are cached and keep their connections tied to the first loop, so a script that made two sync calls on one ModelSpend object failed on the second one with all attempts failed ... Event loop is closed. Sync calls now share one background loop, and a regression test covers repeated route() and stream() calls.
• Not affected: the async API (aroute, astream), the CLI and the local server. Only the sync SDK methods, used more than once on the same object, hit this.

Verified live

• OpenAI structured output (response_schema) on gpt-4.1-nano, strict and non-strict: valid JSON, validation passed, no escalation, with token counts and cost reported. With a non-strict schema the model may use its own wording (it answered "Billing Issue" where the strict enum forced billing); use a strict schema when you need exact labels.
• This test is also what found the bug above.

Not yet verified

• Telling models apart at scale. On the harder set only one of seven models fell short, on one task type with 22 cases.
• OpenAI cached-token reporting, and a comparison of totals with the provider's billing page.
• Gemini and xAI adapters (unit-tested against simulated responses only).
• Concurrent budget enforcement across processes and the dashboard's budget form.
• The Docker code sandbox and automatic price refresh against the real provider pages.
• macOS and Linux with Python 3.10+ only; Windows is untested.

Upgrade

git pull && pip install -e .

Everything else is as in 0.5.0b1: your own evidence bars, your own eval cases, modelspend evidence.

Full changelog: v0.5.0b1...v0.5.0b2

v0.5.0b1 Pre-release

ModelSpend 0.5.0b1 (beta)

ModelSpend shows where your LLM tokens and money go, per project and per agent, enforces budgets before the spend, and can route each task to the lowest-cost model expected to suit it. It uses your own API keys for OpenAI, Anthropic, Google and xAI, stores no prompts, runs locally, and is Apache-2.0.

This release is about proof: you can now measure models on your own cases and say how much evidence you require before ModelSpend picks one. It is still a beta; what was and was not tested live is listed below.

What's new

Your own evidence bars

• modelspend thresholds set extraction --min-n 80 --min-pass-rate 0.95 (also list and remove). Per task category, or * for all, you set how many eval cases a model needs, the pass rate it must show, and whether the bar is judged on the conservative 95% Wilson lower bound (default) or the observed pass rate (--basis observed). --require-evidence means a model is never chosen for that category on its tier prior alone.
• Bars resolve field by field: the category's own setting, then *, then the profile's built-in bar. A setting can only tighten high_assurance's evidence requirement, never loosen it. Settings that cannot be valid (for example min_pass_rate: 95 instead of 0.95) stop with an error rather than being ignored.
• A bar the evidence cannot clear is reported as such, with the number of cases needed, instead of silently failing: a 95% lower-bound bar needs at least 73 cases even if every one passes.

Your own eval cases

• Only id, category, messages and validators are required. Cases can come from one .jsonl file, a folder of them, or <dir>/calibration plus <dir>/heldout.
• eval calibrate --show-failures lists which cases each model failed and its shortened answer (printed to the screen only, never stored).
• modelspend eval check --fixtures my-cases.jsonl costs nothing and catches bad JSON (with file and line), unknown fields or categories, duplicate ids, unknown or misconfigured validators, validators that reject a case's own reference answer or accept its wrong one, sets that share ids, and whether you have enough cases to ever clear your bar. modelspend eval template <category> prints a starter case, and eval calibrate --dataset NAME names the evidence you store.

See who clears your bar

• modelspend evidence (and GET /v1/evidence) lists each model and category with passes out of total, the observed rate, the lower bound and a verdict (clears, below the bar, needs more cases, bar unreachable at this n), and names the cheapest model that clears each category. It uses the router's own functions, so the table and the routing decision cannot disagree. GET /v1/policy now includes your thresholds.

Guide, validators and limits: docs/own-evals.md.

Verified live

• Calibration on a harder case set (OpenAI and Anthropic): the 22 calibration cases of examples/evals/support-tickets-hard on 7 models cost about $0.13. Six models scored 22/22 and cleared the default bar; gpt-4.1-nano scored 19/22 (lower bound 0.67), fell below it, and is no longer chosen for classification. The check, evidence table and a route that selected a model on its measured verdict all worked.
• Held-out comparison: on 8 cases the calibration never saw, ModelSpend matched the premium model's pass rate (8/8) at about 77% lower spend, while the cheapest model missed one (7/8). Eight cases give wide intervals (68-100%), ModelSpend receives your validators and may escalate while the baselines do not, and it did not escalate here.
• From earlier releases: per-project and per-agent reports, hard and soft budgets, the Costs tab, aliases and conversation stickiness, failover on a bad key, and cross-vendor escalation.

Not yet verified

• Telling models apart at scale. Only one of seven models fell short, on one task type with 22 cases. An easy 12-case set separated nothing. Use cases where a cheaper model can plausibly fail (ambiguous inputs, misleading wording, edge cases).
• OpenAI strict structured output (a response_schema call).
• Gemini and xAI adapters (unit-tested against simulated responses only).
• Concurrent budget enforcement across processes and the dashboard's budget form (tested with threads and generated data).
• The Docker code sandbox and automatic price refresh against the real provider pages.
• Totals have not been compared with the providers' own usage or billing pages.
• macOS and Linux with Python 3.10+ only; Windows is untested.

Know before you use it

• Your validators are your definition of "good". ModelSpend measures only what they check, and small numbers of cases give wide intervals. A pass means your checks passed, not that the answer is right.
• One calibration run is one sample. Models are not perfectly repeatable (in a live check the same model scored 19/22, then 20/22 on a re-run); a model close to a bar can flip between runs. Each case counts once.
• Evidence is per model and per task category. It applies when a request is classified into that category (by the rules, or via task_type). Evidence ages: models change under the same name, and ModelSpend does not re-measure on its own. Writing evidence replaces the earlier result for that model and category.
• eval calibrate runs every eligible model on every case, so the cost is roughly cases x models x a small call. --budget is a hard USD ceiling.
• Your thresholds apply under every profile. A lax bar (for example --basis observed on a dozen cases) makes ModelSpend trust thin evidence, which is your call; modelspend thresholds list shows the bar that applies to each category, and where it came from.
• Only calls made through ModelSpend are counted. Your provider's invoice is always the authority on cost. Only Anthropic prices ship; add other providers' models and prices with catalog sync / catalog price or the Pricing tab. See docs/limitations.md and docs/alternatives.md.

Upgrade

cd modelspend && git pull origin main && pip install -e . && modelspend restart

Nothing to migrate. Thresholds live in ~/.modelspend/settings.json and evidence in ~/.modelspend/evidence.json; a running server reads them at start-up, so restart it after changing thresholds.

0.4.0b1 (beta)

ModelSpend 0.4.0b1 (beta)

ModelSpend shows where your LLM tokens and money go, per project and per agent, enforces budgets before the spend, and can route each task to the lowest-cost model expected to suit it. It uses your own API keys for OpenAI, Anthropic, Google and xAI, stores no prompts, runs locally, and is Apache-2.0.

This release is about choice and consistency: you can now pick the model yourself by name, and a multi-turn chat no longer hops between models. It is still a beta; what was and was not tested live is listed below.

What's new

Model aliases: pick the model by the kind of work

• modelspend alias set coding anthropic/claude-sonnet-5-5 and modelspend alias set classify openai/gpt-4.1-nano, then --model classify on the CLI, model="classify" on the local server (any OpenAI client), or pin="classify" in the SDK.
• An alias is a pinned model: exactly that model runs, with no routing, failover or escalation. Tokens, cost and tags are still recorded, so per-project / per-agent reports and budgets work even if you never use automatic routing.
• modelspend alias list tells you if an alias's target is missing from the catalog, has no price, or has no API key. The server has GET /v1/aliases and PUT/DELETE /v1/aliases/{name}; changes apply immediately.

Conversation stickiness: keep a chat on one model

• A request that carries earlier turns now keeps the model chosen for the conversation, instead of being re-routed on every message. Switching models mid-conversation can lose provider-side prompt caching (ModelSpend does not measure this) and changes the model's style partway through.
• It is released, with the reason in the explanation, if that model is no longer suitable for the turn, is denied, or is unavailable. Short follow-ups such as "and the refund?" cannot be classified on their own, so they inherit the task category of the earlier turns instead of jumping to a stronger, more expensive model.
• Results say what happened (sticky: "held" or "released: ..."), and the explanation names the cheaper model you would have got for that turn alone, so the price of stickiness is visible.
• Only a salted hash, a model name, a task category and a time are stored (no content); entries expire after 24 hours. Turn it off per request with sticky=False (or "modelspend": {"sticky": false} on the server) or for everything with MODELSPEND_STICKY=false. Pass a run_id if your client rewrites or compacts its history.

Also since 0.3.0b1

• Fix: the overhead tile and report showed overhead cost as "unknown" when there were no overhead calls; it is now $0.
• Fix: the daily chart repeated the same date on its x-axis for a one-day window.
• Fix: preview --assume-keys --model ... now assumes a key for the pinned model's provider.

Behaviour change to know about

Stickiness is on by default. If you send multi-turn conversations through the server and relied on every turn being routed independently, set MODELSPEND_STICKY=false. Single-turn requests, pinned models and aliases are unaffected.

Verified live

• Aliases and stickiness (OpenAI and Anthropic): an alias ran exactly its model; a second turn stayed on the first turn's model and inherited its category; with stickiness off the same follow-up was routed afresh (to a more expensive tier-2 model, about five times the cost in that one example, which is a single data point, not a benchmark).
• Failover on a bad key: an OpenAI auth error was not retried, Anthropic answered, and the key did not appear in the output.
• Cross-vendor escalation: with an impossible validator, one escalation from OpenAI gpt-4.1-nano to Anthropic claude-sonnet-5-5, then it stopped; tokens and cost reconciled and the first call showed as overhead.
• Reporting, budgets and the Costs tab (from earlier releases): per-project / per-agent reports, hard and soft budgets from the CLI, and the Costs tab on live data.

Not yet verified

• OpenAI strict structured output (a response_schema call).
• Gemini and xAI adapters (unit-tested against simulated responses only).
• Concurrent budget enforcement across processes and the dashboard's budget form (tested with threads and generated data).
• The Docker code sandbox and automatic price refresh against the real provider pages.
• Totals have not been compared with the providers' own usage or billing pages.
• macOS and Linux with Python 3.10+ only; Windows is untested.

Know before you use it

• Stickiness does not weigh the cost of switching against a saving: it switches only when the held model is unusable, never because a cheaper one appeared. A cache-aware comparison is a possible next step. ModelSpend does not send Anthropic cache markers, so for Anthropic the benefit is consistency rather than cache savings.
• Conversations are recognised by their first message (or run_id): two unrelated chats that start with the identical first message share a key.
• Only calls made through ModelSpend are counted. Budget windows are UTC. A hard USD budget refuses a model that has no price. Your provider's invoice is always the authority on cost.
• Only Anthropic prices ship; add other providers' models and prices with catalog sync / catalog price or the Pricing tab. See docs/aliases-and-stickiness.md, docs/cost-attribution.md, docs/limitations.md and docs/alternatives.md.

Upgrade

cd modelspend && git pull origin main && pip install -e . && modelspend restart

Your usage database upgrades itself on first use (a small new table for conversations).

0.3.0b1 (beta)

ModelSpend 0.3.0b1 (beta)

ModelSpend now does two things: it shows where your LLM tokens and money go, per project and per agent, and it routes each task to the lowest-cost model expected to suit it. This release adds the first half (and makes it enforceable with budgets). It uses your own API keys for OpenAI, Anthropic, Google and xAI, stores no prompts, and is Apache-2.0.

Still a beta. This is the first release since 0.1.0b1 (the 0.2 work was never tagged separately, so it is included here). What was and was not tested live is listed below.

What's new

Cost per project and per agent

• Tag any call with project, agent, an optional run_id and up to 8 labels: SDK arguments, CLI flags (--project, --agent, --run-id, --label), MODELSPEND_PROJECT / MODELSPEND_AGENT, or headers on the OpenAI-compatible server (X-ModelSpend-Project, -Agent, -Run, -Labels). Tags are short identifiers (no spaces), so prompt text cannot end up in them. They are attribution, not access control.
• modelspend report --by project,agent,model --since 7d (also CSV and JSON, and GET /v1/usage/report): tokens by type (input, cached, output, reasoning), cost, and overhead, meaning what failed calls, retries, superseded answers after an escalation, and classifier or judge calls cost. Calls on models without a price count their tokens and show cost as partial; nothing is guessed.
• Observe-only mode: pin a model (--model provider/id, pin=, or a non-auto model on the server) to measure an agent without routing it. Adopting it can be as small as a new base_url and two headers.

Dashboard Costs tab (/dashboard#costs)

• A token total, tiles for cost, requests, overhead, cache hit rate and unpriced calls; tokens by group as stacked bars (input, cached, output); tokens per day; a full table; one filter row (window, group by, project) that scopes everything. Light and dark mode, tooltips and keyboard focus, a table view for every chart, and no external resources.

Budgets per project and agent

• modelspend budget set --project shop --period monthly --usd 50 (also list, remove, GET/PUT /v1/budgets, DELETE /v1/budgets/{id}, and a form on the Costs tab). Daily or monthly (UTC), in USD and/or tokens; scope by project, agent, both, or everything.
• Hard: before every provider call (including retries, failovers, escalations, classifier and judge calls, and streams), ModelSpend checks spent so far + calls in flight + this call's worst case. If a limit would be exceeded, nothing is sent and you get HTTP 429 period_budget_exceeded with the amounts and the reset time. The check-and-reserve is one SQLite write transaction, so threads and processes sharing the database cannot jointly overshoot.
• Soft: never refuses; results carry budget_warnings (at 80% and when over, based on actual spend).
• modelspend prune 90d --yes deletes old usage records (budgets are kept).

Also changed

• A "no credits" error (for example OpenAI's credit_balance_exhausted) is now a distinct quota error: it is not retried, and it fails over to another provider if you allowed one.
• The server's model field accepts a catalog model to pin it (before, anything but auto was a 400).
• When an escalation could not run because of a budget, the earlier answer is returned as validation_failed with escalations: 0 (it used to read ok).
• The usage database now runs in WAL mode. You will see usage.db-wal and usage.db-shm files next to it.

Verified live

• Anthropic and OpenAI: routing, token counts and cost (the report's totals matched the providers' reported usage and the catalog-price arithmetic), streaming (OpenAI), and tagged calls in report.
• OpenAI, from the CLI: a hard budget refused a call before it was made (spent 73 + worst case 98 = 171 against a 150-token limit), the refused call was not recorded as spend, and a soft budget let the call run and warned. (OpenAI's own usage page was not compared.)

Not yet verified

• The dashboard Costs tab and budget form were tested in a real browser with generated data, not yet against your own live usage.
• Concurrent budget enforcement is tested with threads on one machine. It is not designed for network file systems or several machines.
• Gemini and xAI adapters: unit-tested against simulated responses only.
• OpenAI strict structured output, failover on a bad key, and cross-vendor escalation.
• The Docker code sandbox and automatic price refresh against the real provider pages.
• macOS and Linux with Python 3.10+ only; Windows is untested.

Know before you use it

• Only calls made through ModelSpend are counted. Traffic that bypasses it is invisible. Importing the providers' own usage data (to reconcile) is planned, not built.
• A hard USD budget refuses to run a model that has no price (it cannot know the cost). Use a token limit for those.
• Budget windows are UTC. Your provider's invoice is always the authority on cost.
• Only Anthropic prices ship; add other providers' models and prices with catalog sync / catalog price or the Pricing tab. See docs/limitations.md, docs/cost-attribution.md and docs/alternatives.md.

Upgrade

cd modelspend && git pull origin main && pip install -e . && modelspend restart

Your usage database upgrades itself on first use (new nullable columns; older rows show as (untagged)).

0.1.0b1 (beta)

ModelSpend 0.1.0b1 (beta)

A cost-aware router for LLM calls. It sends each task to the lowest-cost model expected to suit it, using your own API keys for OpenAI, Anthropic, Google and xAI, and keeps every call, retry and escalation inside a token and dollar budget you set. Python SDK, CLI, a local OpenAI-compatible server (model="auto"), a usage dashboard and an eval harness. Apache-2.0.

This is a beta. It does what is listed below, and it has not been tested in the places listed under "Not yet verified". It does not guarantee the cheapest model or the best answer; it guarantees a documented policy, a budget ceiling and a record of why.

What it does

• Routes by task category, cost and suitability, with three documented profiles (economy, balanced, high assurance).
• Reserves the worst case against your USD and/or token budget before every call, and counts retries, failovers and escalations.
• Token-first accounting from provider-reported usage; cost is shown as unknown when a model has no price.
• Bounded retry and failover for operational errors (including "no credits"), and at most one escalation, only on validator evidence.
• Stores no prompts; keys come from the environment or a 0600 keys file and are scrubbed from errors; the server binds to localhost.
• Guarded price updates: fetch the provider's pricing page, propose changes, verify against the page text, you review and apply.

Verified live (2026-10-12)

• Anthropic: model sync, routing, escalation, token budgets, streaming, dashboard; token counts and cost matched provider-reported usage.
• OpenAI: routing, chat and streaming on gpt-4.1-nano and gpt-5-mini; input-token estimate matched the provider's count; reasoning tokens are reported inside the output count; usage arrives in the stream; cost matches catalog prices. (Not yet compared against OpenAI's billing page.)

Not yet verified

• Gemini and xAI adapters: unit-tested against simulated responses only.
• OpenAI strict structured output, failover on a bad key, cross-vendor escalation.
• The Docker code sandbox (python_tests validator) has not been run against a real Docker daemon.
• Automatic price refresh against the real OpenAI, Google and xAI pages (tested against fake pages and a real browser).
• Only macOS and Linux with Python 3.10+ have been considered; Windows is untested.

Know before you use it

• Only Anthropic prices ship. Add other providers' models and prices with catalog sync / catalog price, or the dashboard's Pricing tab. Check prices against the provider's official page; the tool warns when they get stale but cannot detect changes on its own.
• Until you add eval cases, suitability comes from editorial tier priors (labelled as such in every route explanation).
• Tasks it cannot categorize go to a mid-tier model by default; pass task_type to override.
• See docs/limitations.md and docs/alternatives.md for the full picture and where other tools fit better.

Install

git clone https://github.com/rmencke2/modelspend && cd modelspend
python3 -m venv .venv && source .venv/bin/activate
python -m pip install --upgrade pip && pip install -e .

Then follow the README's setup section. Issues and reports from people running Gemini, xAI or Windows are especially welcome.