Repository navigation
A hard dollar limit per request flow or job with OpenAiChatModel (per-call customHeaders) #7099
domondi1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
In a Spring app, one user action often turns into several
ChatModelcalls: a tool-calling loop, a few parallel summaries, a retry. Token counts are visible after the fact, but nothing stops one job from spending more than it should. Here's one way to put a hard dollar ceiling on a single job using only what Spring AI already has: a differentbase-urland per-requestcustomHeadersonOpenAiChatOptions.Run a small OpenAI-compatible gateway next to the app (it uses your provider key):
Then pass the job's id and budget on each call:
Every call with the same job id draws from one budget. Each call reserves its worst-case cost before it's sent, so parallel calls in one job can't overspend together, and a call that doesn't fit is refused with HTTP 402 before it reaches OpenAI (the call throws).
inferrail work ticket-4812shows what the job cost.Caveats: chat completions only; embeddings should keep their own base URL (
spring.ai.openai.embedding.base-url), since the gateway doesn't serve them; the budget store is a SQLite file on the gateway host, so run one gateway; and the model needs a price in Inferrail (gpt-4o-mini,gpt-4.1-mini,gpt-4.1are built in). I've tested the gateway with the Python and TypeScript OpenAI clients, not a Spring app end to end, so reports are welcome.Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind.
All reactions