Repository navigation
45K+ tokens per chat request??? 💸💸💸 #10471
Replies: 1 comment 1 reply
|
You were right, and the number was worse than your estimate. Thanks for the report, it was a real bug. Fix is up in #10481. What we were actually sendingI measured the payload rather than guessing. Global chat, before:
A single tool, JSON Schema tokenizes badly, roughly 4 characters per token, so your 45K reading is exactly what that adds up to. Your OpenRouter hunch was also correctWe only injected Anthropic What changed
Measured on the benchmark suite, base versus head, same model and cases:
No case regressed. The static character count and the measured token drop agree to within a few percent, so the saving is exactly the fixed overhead you were paying on every "hello". Where your critique landsThe skills point is half right and worth separating. The instructions were already lazy: So your argument applies, one level up from where you aimed it: a small core tool set plus load-on-demand toolsets per domain. That is the right next step and it is now on the list. This PR is the first, largest slice of it. One thing still on the table: tool results accumulating inside a single agent loop land after the last cache breakpoint and are still resent uncached each iteration. Covering those needs a breakpoint on a tool-role message, which OpenRouter does not document, so we did not want to risk breaking every OpenRouter chat on unverified behavior. |


Uh oh!
There was an error while loading. Please reload this page.
I opened the chat in Windmill after connecting it to OpenRouter, and the price of just saying hello to Sonnet 5 and asking the chat what it can do results in 45K+ input tokens for every request.
That's $0.10 just to say hello.
What on Earth are you sending and is that really necessary?
Is this the kind of cost overhead I can expect when an agent is doing an actual task as well?
If so, I am out, right now - this would run me at least $10 a day. That is officially nuts. 🤷♂️
Is this because I'm going through OpenRouter? I know it's slightly more expensive, but I've never seen any chat/agent sending a payload of 45K+ tokens just to bootstrap itself. That is legit crazy.
Even Claude Code doesn't hit more than 7K or so. You've heard of skills, right? You don't need to preload the entire known universe just to greet the user - you can have an agent load required skills when it needs them.
All reactions