Skip to content

v2.5.0

Latest
Latest

Choose a tag to compare

View all tags
@justrunme justrunme released this 19 Oct 17:07
v2.5.0
7d1e514
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Inference budget guard on the gateway chat path (opt-in, INFERENCE_BUDGET_ENABLED).

  • Atomic reserve → execute → settle per tenant or API key: Redis Lua (shared across replicas) or an in-process lock (per process), integer micro-USD
  • 429 when over budget; 503 fail-closed on store down, timeout, corrupt data or malformed policy; X-AI-Budget-Warning: soft_threshold at the soft ratio
  • Each reservation is settled once: actual usage when known, the estimate when usage is unknown or the model may have run, a refund when it never ran
  • ASC-shaped decision envelopes; evidence in docs/EVIDENCE.md / docs/results.json; design in docs/budget-guard.md
  • CI runs multiprocess concurrency tests against a real Redis service container

Limits, on this test suite: the pre-request estimate can differ from actual usage; settlement lags if Redis fails after the upstream call; one Redis, not multi-region billing.

Assets 3
Loading