Asset SLA monitoring

[sryza]

Status: implemented, experimental

The design proposal below resulted in Dagster's FreshnessPolicy feature, which is currently marked experimental.

Here is relevant documentation for this feature:

• API docs: https://docs.dagster.io/_apidocs/assets#dagster.FreshnessPolicy
• Docs on how to trigger arbitrary alerts when freshness policies are violated: https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#freshness-policy-sensors

Original discussion

The aim of this discussion is to gather feedback on a feature we're interested in building. If you're interested in this functionality or functionality that's related, we'd love for you to tell us about it.

Framing

Allow users to attach metadata to asset definitions that indicates when they're expected to be updated by. Dagit shows whether each asset has missed its SLA.

User story

I want to understand when my asset don’t reflects the data I expect them to by the time I expect them to.

Examples:

• "By 9 AM every day, I want asset X to include all the data that arrived by 1 am."
• "By 30 minutes after the hour every hour, I want asset X to contain the partition for the previous hour."
• "Within 30 minutes of when a purchase is recorded in the app DB, it should be reflected in the purchase_stats table."

SLAs and lineage

An asset's SLA is not just relative to its direct parent assets. When we care about the timeliness of an update to an asset, it usually means that we want some event or fact that occurred in the real world to be reflected in the asset by a given time. Often, there's a chain or DAG of assets between the real world and the asset with an SLA. All those assets need to be updated, in order, to meet the SLA.

How do we know whether an asset contains data that arrived by X time?

Option: materialization timing of upstream assets

If asset X depends on asset Y, and the graph or op that materializes asset X starts running at time T_0 and completes at time T_1, then we can say that asset X reflects all the data that's reflected in asset Y by time T_0.

Option: partition existence

If an asset is daily-partitioned, and there exists a partition for day D, we might assume that it contains all the data for upstream assets that arrived during D.

Solution proposal

Python API

"By 9 AM every day, I want asset X to include all the data that arrived by 1 am."

@asset(sla=SLA(materialized_by="0 9 * * *", data_arrived_by="0 1 * * *"))
def asset_x(asset1):
    ...

"By 30 minutes after the hour every hour, I want asset X to contain the partition for the previous hour."

@asset(
    partitions_def=HourlyPartitionsDefinition(start_date=...),
    sla=SLA(materialized_by="30 * * * *")
)
def asset_x(asset1):
    ...

Dagit

We could show a view with a row per asset, that reflects whether the asset hit its SLA and how overdue it is if not.

You could click into an asset to understand how its lineage affects its SLA - i.e. whether there are any upstream assets that need to be rematerialized for it to hit its SLA.

Future work

• Use SLAs for scheduling
• Show historical performance against SLAs
• Send emails / slack messages / etc. when assets miss their SLAs

[andreqaugusto]

I love the idea and would like to see it implemented. Just want to note that the particular point of using SLAs for scheduling resonates with this post by Benn Stancil.

[yeachan153]

This sounds great! One thing that would be good to have on top would be the ability to send alerts based on missed SLAs.

[abkfenris]

Cross referencing with #13102

[colin-mills]

This sounds like exactly what we need. I would also advocate allowing for setting SLA and SLO. We have our SLAs defined more leniently than our SLOs. When an SLO is violated that is a much bigger deal and we would prefer to have the native Dagster ability to track both.

[sryza]

@colin-mills not sure if you saw, but we ended up implementing the functionality above as the FreshnessPolicy feature (still experimental).

Here is some relevant documentation:

• API docs: https://docs.dagster.io/_apidocs/assets#dagster.FreshnessPolicy
• Docs on how to trigger arbitrary alerts when freshness policies are violated: https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#freshness-policy-sensors

[colin-mills]

Thanks, this looks great.

[colin-mills]

Is there a recommendation for how we could use these freshness policies with an SLA and an SLO? In our current setup, we are orchestrating dataset creation for various teams who have both of these values defined in the dataset definitions. In our code, we would like to be able to take different actions when an SLA (less serious) is violated than when an SLO is violated (more serious).

SLA deadline is always sooner than the SLO deadline, but for any given dataset the time between SLA and SLO violation may differ. For example:

• dataset A: SLA: 12 (hours); SLO: 16 (hours)
• dataset B: SLA: 36 (hours); SLO 54 (hours)