RFC: per-asset "backfill strategy" that determines whether / how much to coalesce

[sryza]

Dagster offers the ability to do "single-run range backfills", i.e. launch a run that backfills multiple contiguous partitions. Currently, you opt in to this functionality by flipping a checkbox in the modal that you use to launch the backfill. This RFC proposes:

1. Allowing users to specify how they want their asset to be backfilled, on the asset definition
2. Allowing values in between "every partition in its own run" and "all partitions in a single run" - e.g. "10 partitions per run"

Motivation

Why (1):

• For any asset, there's typically a "right" way to backfill it. E.g. most Snowflake assets should handle large ranges in a single run, most Pandas assets should have a run per partition. Putting the choice in code makes it so that users don't need to make the choice every time they launch their backfill, and reduces the risk of actually getting it wrong.
• This would allow the backfill engine to e.g. execute a backfill that fills in a Pandas asset one partition at a time and then, after, fills in a Snowflake asset with a single run.
• Auto-materialize could use it to coalesce multiple partitions into single runs.

Why (2):

• Even with a powerful distributed computation engine like Snowflake or Spark, backfilling a really big table in a single run can be unwieldy. Two data points for this include my personal experience and feedback from a large user.

Proposal

20 partitions per backfill:

@asset(backfill_strategy=BackfillStrategy(max_partitions_per_run=20))
def asset1():
    ...

All partitions in a single run:

@asset(backfill_strategy=BackfillStrategy.single_run())
def asset1():
    ...

One backfill strategy per multi-asset:

@multi_asset(backfill_strategy=BackfillStrategy(max_partitions_per_run=20))
def asset2():
    ...

Open questions:

• Should we remove the toggle in the backfill dialog and make backfill strategy the only way of controlling this?
• Should this affect runs kicked off by auto-materialize? I think yes.
• If this affects runs kicked off by auto-materialize, does this mean we should rename max_materializations_per_minute to max_partitions_per_minute? Otherwise, we'd risk backfilling large amounts of data in a single run.
• Other names to consider:
  • Backfill policy
  • PartitionedRunStrategy

[ruizh22]

@sryza Thanks for putting this together!

[bmarcj]

See also #14999 for motivation on the GUI users side.