Configuring partition-aware freshness policy in Dagster

[ignacyklimont]

Hello!

I would like to start using freshness policies for our deployments - particularly CRON ones - however one thing that bothers me is that it seems fully partition-agnostic, meaning that it simply requires any partition to materialise within the time window, not necessarily the latest one.

The issue for my use case is that we want to use it to track monthly, weekly and daily workloads, and of course the goal is to monitor and spot any latest partitions that are missing. We want to avoid situation where e.g. backfilling some old partition or re-running it would automatically set the asset to "fresh", despite the fact that the latest partition is in fact missing.

How can this be tackled? Would partition-aware freshness policies be supported in future in Dagster? If so, when? Is there a better method/tool in Dagster to monitor scheduled workloads with rigid frequency (daily, weekly, monthly, quarterly) for staleness and incomplete/missing/failed runs?

So far we had a custom sensor to handle that, but that in itself was quite buggy. We would prefer to use a Dagster-native solution; the freshness policies looks like an almost perfect tool, albeit the fact that it has no visibility which partition actually ran and whether it was the latest makes it also not ideal for the job.

Thanks for any tips and help!

[yhay81]

Your reading of the new policy is correct. The current evaluator is asset-level, not partition-level: it loads last_materialization for the asset and compares only that event's timestamp with the cron window. It never inspects the materialization's partition key: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/freshness_evaluator.py#L88-L143

So a recent backfill of an old partition can indeed turn the new FreshnessPolicy.cron(...) green. I would not use that policy as the sole signal for “the expected latest partition exists” yet.

There is, however, a Dagster-native check that implements exactly your required semantics: build_time_partition_freshness_checks. At each deadline it computes the most recent completed time-window partition and looks up a materialization for that specific partition key. A backfill of another key cannot satisfy it: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_factories/freshness_checks/time_partition.py#L115-L176

For example:

import dagster as dg

daily_checks = dg.build_time_partition_freshness_checks(
    assets=[daily_asset],
    deadline_cron="0 9 * * *",  # expected previous daily partition by 09:00
    timezone="UTC",
)

weekly_checks = dg.build_time_partition_freshness_checks(
    assets=[weekly_asset],
    deadline_cron="0 9 * * 1",  # expected previous weekly partition by Mon 09:00
    timezone="UTC",
)

freshness_checks = [*daily_checks, *weekly_checks]
freshness_sensor = dg.build_sensor_for_freshness_checks(
    freshness_checks=freshness_checks,
)

defs = dg.Definitions(
    assets=[daily_asset, weekly_asset],
    asset_checks=freshness_checks,
    sensors=[freshness_sensor],
 )

Use one check group/deadline per cadence. It works for TimeWindowPartitionsDefinition subclasses, so daily/weekly/monthly (and a custom time-window definition for quarterly) fit. A missing partition or a failed run produces no materialization record for the expected key, so the check stays failed; an old-partition retry does not change it.

The caveat is API direction: build_time_partition_freshness_checks is currently marked superseded in favor of attaching FreshnessPolicy objects. But Dagster's current freshness documentation also explicitly says that freshness checks “will continue to be supported for the foreseeable future,” while the new policies are under active development and have feature gaps: https://docs.dagster.io/guides/observe/asset-freshness-policies#relationship-to-freshness-checks

Therefore my migration path would be:

1. keep the partition-aware check/sensor for the correctness alert today;
2. optionally add the new policy for the general asset-level signal/UI;
3. move to the new policy only after its evaluator selects the expected partition key.

I cannot find a public commitment or release date for partition-aware evaluation in the new policy, so only the maintainers can give an ETA. The current published “Future enhancements” list does not promise it yet.

[ignacyklimont]

Hello,

Thanks for the answer, this is very informative. It is surprising that the freshness checks are already superseded, while no alternative is available via the latest freshness policy.

Furthermore, checking availability of the latest partition for schedule-based workloads seems like a standard and basic logic for freshness policy. Nonetheless, I will indeed consider adding freshness checks for now, so that both would be evaluated (given freshness policy is the future-oriented API). Hopefully this functionality will be added to freshness policy soon.