Repository navigation
A hard dollar cap per Dagster run for assets that call OpenAI #34254
domondi1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
dagster-openairecords token usage on each materialization, which is great after the fact. What it can't do is stop a run partway: an asset that summarizes 5,000 rows, a backfill over many partitions, or a thread pool of parallel calls will spend whatever it spends, and you see the total once it's done.Here's a way to put a hard dollar limit on one run without changing
dagster-openai. PointOpenAIResourceat an OpenAI-compatible gateway that enforces a per-run budget, and send the run's id and budget as headers on each call. Usingcontext.run_idas the key means every op and asset in the run, across processes, draws from the same budget.Run the gateway next to your code location (it uses your provider key):
After the run:
shows how many calls the run made and what they cost.
Each call reserves its worst-case cost before it's sent, so the parallel calls can't overspend together. A call that doesn't fit gets HTTP 402 before it reaches OpenAI and raises
openai.APIStatusErrorwithstatus_code == 402. If you don't catch it, the asset fails and the run stops there. The headers aren't forwarded to OpenAI. I usedextra_headersper call rather thanclient.with_options(...)because it keepsget_client(context)'s usage metadata (openai.<model>.calls, tokens) on the materialization.What I tested: dagster 1.13.25, dagster-openai 0.29.25, inferrail 0.4.12, with
materialize()and a local stub upstream standing in for OpenAI. Ten parallel calls with a generous budget all succeeded. With a tight budget, 3 went through, 7 were refused before reaching the upstream, and the materialization still carried the usage metadata. I haven't run it in a deployed Dagster instance against the real API, so reports are welcome.Caveats: chat completions only (keep embeddings on a separate resource without
base_url); the budget store is a SQLite file on the gateway host, so run one gateway that all run workers can reach; and the model needs a price in Inferrail (gpt-4o-mini,gpt-4.1-mini,gpt-4.1built in,inferrail modelslists others, and you can add your own prices). If you'd rather budget per partition than per run, usef"{context.run_id}:{context.partition_key}"as the id.Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind. A
default_headersfield onOpenAIResourcewould make this a config-only setup for any proxy, and I'm happy to send a PR if that's welcome.All reactions