partitioned asset downstream of non-partitioned asset

[brandybobandyistaken]

What is the expected behavior when I have a monthly partitioned asset downstream of a non-partitioned daily asset?

The purpose of the non-partitioned daily asset is to ensure that the data that the monthly partitioned asset relies on is up to date.

Will the monthly partitioned asset wait for the daily non-partitioned asset to run (desired behavior), or is there a chance that the monthly partitioned asset sees yesterdays materialization of the daily non-partitioned asset and kicks off, since the daily asset is non-partitioned (undesirable behavior).

I'm using automaterialization.

The flow is essentially:

1. automaterialize daily non-partitioned root asset with cron rule
2. other downstream daily non-partitioned assets kick off with eager automaterialize policy
3. monthly partitioned assets kick off once steps 1 and 2 are done

Thanks!

[garethbrickman]

The expected behavior is that the monthly partitioned asset will wait for the daily non-partitioned asset to run before it kicks off. This is because the auto-materialization system in Dagster is designed to ensure that downstream assets are only materialized when their upstream dependencies are up to date.

In your case, you have a non-partitioned daily asset that is set to auto-materialize with a cron rule. This means that the asset will be materialized based on the schedule defined by the cron expression. Downstream non-partitioned assets with an eager auto-materialize policy will then attempt to materialize as soon as their upstream dependencies (including the daily non-partitioned root asset) have been materialized.

For the monthly partitioned assets, the behavior will depend on the auto-materialize policy you have set for them. If they also have an eager auto-materialize policy, they will wait until all their upstream dependencies, including the daily non-partitioned assets, have been materialized for the current period before attempting to materialize.

By default, the eager policy will only materialize an asset if it's missing or one of its parents update. This means that an unpartitioned root asset will only get auto-materialized a single time, as it has no parents which can update. However, you can use the materialize_on_cron rule to ensure that the root asset is re-materialized on a regular basis according to a cron schedule.

Here's an example from the documentation that shows how to set up a materialize_on_cron policy:

from dagster import AutoMaterializePolicy, AutoMaterializeRule, asset

materialize_on_cron_policy = AutoMaterializePolicy.eager().with_rules(
    AutoMaterializeRule.materialize_on_cron("0 9 * * *", timezone="US/Central"),
)

@asset(auto_materialize_policy=materialize_on_cron_policy)
def root_asset():
    ...

In this example, the root_asset will be materialized every day at 9 AM Central Time. If your monthly partitioned assets depend on this root_asset, they will only materialize once the root_asset has been updated according to its cron schedule.

[brandybobandyistaken]

This is a fantastic reply, and the behavior is what I was hoping for. Thank you very much!

[garethbrickman]

You're most welcome! Check out our docs for Auto-Materializing Assets to see the latest policies available.