Capping what one batch of llm_client image captions can cost

[domondi1]

With llm_client set, MarkItDown sends one vision request per image: every picture in a PPTX and every image file you convert. Converting a folder of slide decks is one job that can turn into hundreds of image calls, and since MarkItDown doesn't set max_tokens on those requests, there's no natural ceiling on what one batch costs. If a run goes sideways (a huge deck, a retry loop in your script), you find out on the invoice.

llm_client is just an OpenAI client, so you can put a hard dollar limit on a batch without changing MarkItDown: point the client at an OpenAI-compatible gateway that enforces a per-job budget, and tag every request with the batch's id and budget.

pip install "markitdown[all]" inferrail
export OPENAI_API_KEY=sk-...

import inferrail
from openai import OpenAI
from markitdown import MarkItDown

base_url = inferrail.start()  # gateway on a background thread in this process

def markitdown_for_batch(batch_id: str, budget_usd: str) -> MarkItDown:
    client = OpenAI(base_url=base_url, api_key="unused", default_headers={
        "X-Inferrail-Attribute-Work-Id": batch_id,
        "X-Inferrail-Budget-Usd": budget_usd,
    })
    return MarkItDown(llm_client=client, llm_model="gpt-4o-mini")

md = markitdown_for_batch("decks-2026-10-03", "2.00")
for path in paths:
    result = md.convert(path)

Afterwards:

inferrail work decks-2026-10-03

shows how many caption calls the batch made and what they cost.

What happens when the budget runs out depends on the converter, which is worth knowing even if you don't use a gateway:

• PPTX: MarkItDown already catches caption errors, so slides past the budget fall back to the image's alt text and the conversion still succeeds. In my test deck of 8 images with a tight budget, 5 got LLM captions and 3 fell back.
• Image files: the error isn't swallowed. convert() raises FileConversionException wrapping the 402, so catch it in your loop if you'd rather keep going without a description.

Each request reserves its worst-case cost before it's sent and is refused with HTTP 402 before it reaches OpenAI if the batch can't cover it. Because MarkItDown doesn't pass max_tokens, the reservation assumes a 4,096-token reply (plus a fixed allowance per image), so the last request or two before the cap may be refused a bit early. Unused reservation is released as each caption comes back, and the batch total is based on real usage. The two headers aren't forwarded to OpenAI.

What I tested: markitdown 0.1.8 and inferrail 0.4.12, with a local stub upstream standing in for OpenAI. I converted an 8-image PPTX and a PNG with a generous budget and with a tight one. I haven't done a full real-API run yet. Caveats: chat completions only; the model needs a price in Inferrail (gpt-4o-mini, gpt-4.1-mini, gpt-4.1 are built in, inferrail models lists others, and you can add your own); inferrail.start() keeps its budget in a local SQLite file, so for several worker processes run one shared inferrail serve instead.

Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind.