How to avoid creating a run for each partition when executing a backfill

[dagsir[bot]]

I’m backfilling some data, and it seems that Dagster Cloud spins up a new instance for each run. Is there a way around that? I have 60,000 runs to backfill, so spinning up an instance for each will take forever.

---

The question was originally asked in Dagster Slack.

[sryza]

For asset backfills, Dagster can let you materialize many partitions in a single run: https://docs.dagster.io/concepts/partitions-schedules-sensors/backfills#launching-single-run-backfills-using-backfill-policies-.

Would that help here? Or are you specifically looking for the ability to have a run per partition, but make the individual runs more lightweight?

[emigal]

Oh, that's fantastic - I'll try that. IMO this should be the default behavior, btw. I can't think of many instances when you actually need to spin new instances for backfilling.

[bmarcj]

To make the opposite case, I don't think it should be the default! Dagsters io model typically means loading all the data and producing all partitions all at once. That's frequently unworkable.

If the IO model defaulted to more iterative model (which can be done) it would make sense, but that's much more complicated to work with.

[emigal]

That's a great point, however something like Snowflake could handle that kind of load, no? In any case, I like the iterative idea - maybe that can be a param for single_run().

[bmarcj]

Yes, or dbt incremental models. But it's a stretch to assume all assets are offloading io/compute.

It maybe should be the default for dbt assets, because the alternative of passing down partition keys to dbt requires more hoop jumping.