A hard dollar limit per crawl job for LLMExtractionStrategy (base_url + extra_headers)

[domondi1]

LLMExtractionStrategy makes one model call per chunk, so one crawl over many pages can turn into hundreds of paid calls, and result.token_usage tells you the count afterwards. If you want a hard dollar ceiling per crawl job instead, you can do it with what LLMConfig and extra_args already accept: point base_url at a local gateway and send a job id plus a budget as headers. I maintain the gateway (Inferrail, open source), so weigh that accordingly.

pip install crawl4ai inferrail
export OPENAI_API_KEY=sk-...
inferrail serve --quickstart --app-mode      # gateway on http://127.0.0.1:8000

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig, LLMExtractionStrategy

def extraction_for_job(job_id: str, budget_usd: str) -> LLMExtractionStrategy:
    return LLMExtractionStrategy(
        llm_config=LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token="unused",                    # the real key stays in the gateway
            base_url="http://127.0.0.1:8000/v1",
        ),
        instruction="List the product names and prices on this page.",
        extra_args={
            "max_tokens": 800,
            "extra_headers": {
                "X-Inferrail-Attribute-Work-Id": job_id,   # one id for the whole crawl
                "X-Inferrail-Budget-Usd": budget_usd,      # its dollar ceiling
            },
        },
    )

config = CrawlerRunConfig(extraction_strategy=extraction_for_job("catalog-2026-10-06", "2.00"))
async with AsyncWebCrawler() as crawler:
    results = await crawler.arun_many(urls, config=config)

Every chunk of every page in that crawl carries the same id, so they share the $2.00, including chunks extracted in parallel. A call that would push the job past it gets HTTP 402 before it reaches OpenAI. Then:

inferrail work catalog-2026-10-06      # calls, tokens and dollars for that crawl

What I checked (crawl4ai 0.9.4, inferrail 0.4.15, calling the strategy's run() on a 10-chunk page against a stand-in OpenAI endpoint with fixed token counts, so no browser and no real spend):

• Enough budget: 10 chunks became 10 calls, all under one id, cost and tokens recorded per call.
• Budget that runs out partway: 3 chunks ran, the other 7 were refused before reaching the upstream (reservations hold under crawl4ai's threaded chunking), and the crawl still returned: refused chunks come back as error blocks rather than an exception.
• crawl4ai only retries RateLimitError, so a refused chunk is tried once, not in a loop.

Caveats:

• A dollar budget needs a price for the model. Most OpenAI and Anthropic models are built in (inferrail models); for other providers you add a price in the gateway config. An unpriced model is refused with a clear error, not guessed.
• Per-page cost works the same way: create a strategy per URL with Work-Id set to the URL or page id.
• The gateway records usage and cost per call, never page content or prompts.