Repository navigation
Putting a hard dollar cap on one evaluate() run #3416
domondi1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
A single
evaluate()fans out into metrics times test cases judge calls, run concurrently by default. On a big dataset with a few metrics that's a lot of model calls, and there's no single place that stops the run at a dollar amount. If a metric loops or the dataset is bigger than expected, you find out on the provider bill.GPTModeltakes the same kwargs as the OpenAI client, so you can cap a whole run without changing DeepEval: point the judge at an OpenAI-compatible gateway that enforces a per-run budget, and tag every judge call with the run's id and budget.pip install deepeval inferrail export OPENAI_API_KEY=sk-...Afterwards:
shows how many judge calls the run made and what they cost.
Each call reserves its worst-case cost before it's sent, so the concurrent metric calls can't overspend together. Once the run's budget can't cover another call, it's refused with HTTP 402 before it reaches OpenAI. With
ignore_errors=Truein your eval config, the refused metric is recorded as an error and the run finishes; otherwise it raises. DeepEval's Tenacity retry only covers 5xx/429, so a 402 isn't retried. The two headers aren't forwarded to OpenAI.One thing worth flagging: G-Eval scores from
top_logprobs, so this needs a gateway that forwardslogprobs. Inferrail added that in 0.4.13 (earlier versions rejected the request with a 400). If you're on an older build, upgrade.What I tested: deepeval 4.2.8 and inferrail 0.4.13, with a local stub upstream standing in for OpenAI, 10 test cases with G-Eval + AnswerRelevancy at
max_concurrent=10. With a generous budget all 20 metric results succeeded; with a tight one, the judge calls past the budget were refused before reaching the upstream and the run's recorded total stayed under the cap. I haven't done a full real-API run, so reports are welcome.Caveats: chat completions only; the model needs a price in Inferrail (
gpt-4o-mini,gpt-4.1-mini,gpt-4.1built in,inferrail modelslists others, and you can add your own);inferrail.start()keeps its budget in a local SQLite file, so for several worker processes run one sharedinferrail serveinstead.Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind.
All reactions