Skip to content

v0.2.0: per-key limits, budgets, and cost estimates

Pre-release
Pre-release

Choose a tag to compare

@yosephfetene yosephfetene released this 06 Nov 08:56
· 6 commits to main since this release
476d40b

ModelSwitch can now be shared with other users: each API key has its own
limits, and you can see and cap what it spends.

What's new

  • Rate and concurrency limits: each key is limited in how fast it can
    send requests and how many it can have running at once.
  • Monthly budgets: give a key a limit in USD, and it stops at about
    that amount until the next month.
  • Cost estimates: set versioned prices, and each request records an
    estimated cost. Old estimates stay the same when prices change.
  • Key expiry and live keys: keys can stop working on a set date, and
    ms_live_ keys are easy to tell apart from ms_dev_ keys.

Good to know

  • Budgets are firm, not exact. Requests already running can take a key
    slightly over its limit.
  • Costs are estimates from token counts the provider reported. Failed
    attempts aren't counted, so your real bill can be higher.
  • Keys have no budget unless you set one.
  • A key with a budget can only use models that have a price.
  • Rate and concurrency limits are counted per server.

Upgrading

Run modelswitch migrate before starting this version.

This is a pre-1.0 release, so details may change.