Metering tokens per customer in Hono, including streamed responses #5600
supertj
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
If your LLM-backed API sets quotas in tokens, counting requests isn't enough. The handler only knows the number at the end, and with server-sent events the end comes after the handler has already returned. Here's how to count tokens per customer in Hono with the UsageRail middleware, and the parts that are easy to get wrong.
The word loop stands in for a model stream. What happens to a request:
429 quota_exceededand the handler never runs.routePath(c). That's why usage lands under/v1/chat/:modelinstead of one row per model name.track()are added when the stream ends or the client disconnects.The gotchas:
track()replaces, it doesn't add. Callingtrack(1)per chunk records the whole stream as 1 unit. Pass the running total.c.get("usageRail")only exists where the middleware runs. TheVariablestype says it's always there, so a handler outside/v1/*that calls.track()compiles fine and throws at runtime.flushOnExit, counts that haven't gone out when the process stops are lost.fileSnapshotStoresaves the key list so a restart can check keys before its first sync.serverless: trueand enablenodejs_compat. Reporting then runs inc.executionCtx.waitUntilafter each response instead of on a timer, and the engine is created on the first request, because Workers don't allow generating random values at module top level.Disclosure: I work on UsageRail. The Hono setup and options are at https://usagerail.com/docs/hono
All reactions