Repository navigation
v0.2.0: per-key limits, budgets, and cost estimates
Pre-release
Pre-release
ModelSwitch can now be shared with other users: each API key has its own
limits, and you can see and cap what it spends.
What's new
- Rate and concurrency limits: each key is limited in how fast it can
send requests and how many it can have running at once. - Monthly budgets: give a key a limit in USD, and it stops at about
that amount until the next month. - Cost estimates: set versioned prices, and each request records an
estimated cost. Old estimates stay the same when prices change. - Key expiry and live keys: keys can stop working on a set date, and
ms_live_keys are easy to tell apart fromms_dev_keys.
Good to know
- Budgets are firm, not exact. Requests already running can take a key
slightly over its limit. - Costs are estimates from token counts the provider reported. Failed
attempts aren't counted, so your real bill can be higher. - Keys have no budget unless you set one.
- A key with a budget can only use models that have a price.
- Rate and concurrency limits are counted per server.
Upgrading
Run modelswitch migrate before starting this version.
This is a pre-1.0 release, so details may change.