Repository navigation
A hard dollar limit on one RAG-Anything ingestion job (text + vision calls) #378
domondi1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Processing a batch of PDFs makes a lot of model calls: entity extraction per chunk, plus a vision call for every image, table and equation the parser finds. With
llm_model_max_asyncseveral run at once, so the cost of oneprocess_document_completerun on a big folder is hard to predict. Here's one way to give each ingestion job a hard dollar budget without changing RAG-Anything.openai_complete_if_cachepasses extra kwargs tochat.completions.create, so the model functions you already write can send the job's id and budget asextra_headersto a small OpenAI-compatible gateway running in the same process:Each call reserves its worst-case cost before it's sent (images count as a fixed 3,000-token estimate; the real cost comes from the provider's reported usage), so concurrent calls can't all spend the same remaining money, and a call that doesn't fit is refused with a 402 before it reaches OpenAI. LightRAG doesn't retry a 402, so the job stops quickly.
inferrail work ingest-2026-10-03shows what the job cost.Caveats: needs
inferrail0.4.12 or later for the image calls; setmax_tokens(the reservation uses it); leaveOPENAI_API_BASEunset so embeddings keep going to OpenAI; cached responses don't count.gpt-4o-miniandgpt-4ohave built-in prices.Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind.
All reactions