Repository navigation
Putting a dollar cap on one PaperQA question #1360
domondi1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
One
askcan make a lot of LLM calls: the agent's tool loop, plus a summary call for every evidence chunk, many of them in parallel. On a large paper directory the cost of a single question varies a lot. Here's one way to give each question a hard dollar budget using only the LiteLLM config PaperQA already takes.llm_configis a LiteLLM Routermodel_list, andlitellm_paramsacceptsextra_headers, so every call for a question can carry its id and budget to a small OpenAI-compatible gateway running in the same process:Each call reserves its worst-case cost before it's sent, so the parallel summary calls can't all spend the same remaining money, and a call that doesn't fit is refused with a 402 before it reaches OpenAI (LiteLLM raises it as an error).
inferrail work q-001shows what the question cost. I checked the Router path itself: withextra_headersinlitellm_params, 6 concurrent calls on a budget with room for 2 got 2 answered and 4 refused, and the headers weren't forwarded to the provider.Caveats: embeddings aren't routed through the gateway (it only handles chat completions), so they aren't counted; set
max_tokenssince the reservation is based on it; and the model needs a price in Inferrail (gpt-4o-miniandgpt-4.1are built in). How PaperQA's agent reacts to the refused call (stop vs. continue with what it has) I haven't tested end to end.Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind.
All reactions