Skip to content

v0.1.2 — streamed responses accrue spend

Choose a tag to compare

@github-actions github-actions released this 10 Jul 13:00
· 108 commits to main since this release

budget_usd now sees streaming traffic. Most agent harnesses stream by default, so until this release the dollar cap was blind to nearly all real traffic — auth, rate limits and the loop breaker covered streams, but spend did not.

No config change is required. This is not a breaking release.

What changed

A streamed response is forwarded byte-for-byte, so the only chance to learn what it cost is to watch it go past. Stoke now reads the provider's own usage report out of the live stream and charges the key when the stream ends — or when the client disconnects mid-answer, which cost the same. Both wire formats are supported: OpenAI reports on a final frame, Anthropic reports input tokens up front and a running output count.

  • Metered providers are asked to report usage (stream_options.include_usage), which adds one final frame to their SSE stream. That is standard OpenAI behaviour, and clients that follow the spec ignore it.
  • Your own hardware is not asked. Providers on local and remote tiers cost electricity, not tokens. Their requests and their streams are byte-for-byte what they always were.
  • A provider that reports nothing is billed from an estimate, not from $0, and the estimate is surfaced separately as estimated_usd in /v1/budget. A provider that hides its usage must not thereby switch the cap off. Treat that number as a guess.

Also fixed

  • A streamed vote race priced the winner against the model the caller asked for rather than the one that actually served — an undercharge, and a silent $0 when the requested model was unpriced.
  • stream_race_models raced every candidate in parallel even on a metered provider, paying for N answers to use one while metering only the winner. Metered providers are now tried in order, which is what its sibling already did and documented.
  • Anthropic reveals input tokens in message_start, before any output exists, so a stream cut short produced a usage object that looked measured. Only the frame carrying the final tally counts as a measurement now; anything else is an estimate and says so.

Known limits

  • The cap is a ceiling you can cross once, not a hard stop. It is checked when a request is admitted and charged when the stream completes. A single very long response overshoots it, and concurrent requests on one key are each admitted against a spend figure that does not yet include the others still in flight. There is no reservation against in-flight spend. Bound it with rate_limit_rpm, or give each agent its own key.
  • Prices are yours to keep current; nothing warns you when a provider changes its rates. stoke-cli pricing shows what the gateway believes.
  • Spend is in-memory and resets on restart.

Install

curl -sSf https://stokegate.com/install | sh