backfill_policy not working?

[jPinhao-Rover]

In our team we have some very large models which we'd like to be able to separate into chunks and do a rolling backfill of data across multiple jobs instead of a single long-running job.

What we'd like to achieve:

• Configure a partitioned asset at a sensible grain, eg. daily
• Be able to run a full historical backfill for said asset
• Have the historical backfill materialise a fixed number of partitions in a single step

In the backfill docs it references that you can specify a backfill strategy, and dagster provides out of the box:

• BackfillPolicy.single_run to configure backfills to run as a single job
• BackfillPolicy.multi_run which will break down the backfill into chunks of max_partitions_per_run partitions per run

The multi_run config sounds excatly like what we'd want to explore, as we could configure backfills to materialise "X days" worth of partitions per job.

However setting the backfill_policy on assets doesn't seem to have any effect. Whether single_run / multi_run, and independently of what max_partitions_per_run setting we configure, backfills seems to launch 1 run per partition. If single_run worked we could also chunk it up within the asset execution, but ideally would be able to have several runs with a scoped number of partitions.

Tested on dagster versions 1.5.4 and 1.5.9.

Not sure if backfill_policy is just broken, or if we need to configure anything else within our assets or instance to make this work? This is how the assets are configured:

PARTITION_DEFINITION = DailyPartitionsDefinition(
   start_date=datetime(2021, 11, 17),
   end_date=datetime(2024, 11, 14),
   timezone="Etc/UTC",
)

@asset(
   partitions_def=PARTITION_DEFINITION,
   group_name="spike_partitions",
   compute_kind="python",
   backfill_policy=BackfillPolicy.multi_run(max_partitions_per_run=15),
)
def test_standalone_asset(context):
   start, end = context.partition_time_window
   context.log.info(
       f"Running test_standalone_asset for partitions: [{start.strftime('%Y-%m-%d')} , {end.strftime('%Y-%m-%d')}]"
   )


@asset(
   partitions_def=PARTITION_DEFINITION,
   group_name="spike_partitions",
   backfill_policy=BackfillPolicy.single_run(),
)
def test_standalone_asset_single_backfill(context):
   start, end = context.partition_time_window
   context.log.info(
       f"Running test_standalone_asset_single_backfill for partitions: [{start.strftime('%Y-%m-%d')} , {end.strftime('%Y-%m-%d')}]"
   )

test_partitions_job = define_asset_job(
   name="test_partitions_job",
   selection=AssetSelection.groups("spike_partitions")
)

[sryza]

Hi @jPinhao-Rover - I believe you're hitting a current known limitation, which is that backfill policies don't apply to backfills launched from the job page. They should work if you launch the backfills from the asset graph or the asset details page.

Here are two relevant issues:

• Backfill policies for jobs #17404
• In job backfill UI, make it clear that backfill policies are not respected #17665

[toddy86]

Ahhhh. I got tricked by this (again).

@sryza, it would be really good that until such point as single run backfills are possible from jobs that there is a warning on the backfill / materialization dialogue that says something to the effect of "Warning: Your asset has a single run backfill policy set, but this will not be respected as you are attempting to run a backfill from the job page. Please backfill from x, y or z to ensure the backfill is completed in a single run".

[sryza]

@smackesey - is this something you were working on?

[jPinhao-Rover]

Thanks @sryza ! Very confusing, particularly as triggering an individual asset from the job UI seems to trigger a job run that executes that asset, rather than asset run, so the backfill policy is not respected. Asked in one of the linked posts as well, but in case you happen to know:

Is there a way to prevent a job from triggering if it is a backfill?

[sryza]

Alas, there is currently not a way to do that.