Alerting and monitoring with Dagster

[yuhan]

This discussion is for folks to share how you handle alerting and monitoring with Dagster.

Question was originally asked in Dagster Slack. Reposting it here for posterity and discoverability.

"""
How do folks handle metrics, dashboards, and alerting related to their Dagster pipelines?

Examples:

• I want to get an alert if the pipeline doesn't finish successfully.
  I want a trend of overall pipeline duration over days (I think Dagster does this), and for different multi-op portions of the DAG (some notional 'stage 1', 'stage 2' of the pipeline that may map to a subgraph but is larger than an op).
• I want to report a metric if a particular op encounters some condition (such as no response from an API), and able to get an alert if too many ops in a fan-out hit that condition, sliced by some attribute (green widgets had 20 errors, blue widgets had 4k errors).
• One particular op had 500 ERROR-level log lines. Find the op, group the log lines by error message.

How much of this can Dagster do natively? What services do folks use for other pieces -- DataDog, ELK, other?
"""

[markfickett]

Thanks @yuhan for copying this over! Some more details I've found as I've done more research:

• Most observability platforms are designed around webapps, which means their builtin instrumentation uses trace context forwarding via HTTP headers (for which there's a W3C spec). Shared trace contexts are the key to linking work done between different microservices, or different processes. But, since Dagster's @op s don't do that forwarding, it's hard to trace any work that crosses an op boundary without custom handling of trace context. More general distributed tracing background: https://newrelic.com/resources/ebooks/quick-introduction-distributed-tracing . And about manually forwarding trace context in OpenTelemetry: https://opentelemetry.io/docs/instrumentation/python/cookbook/#manually-setting-span-context .
• Several observability platforms (New Relic, Honeycomb, Lightstep, Jaeger, likely others) support OpenTelemetry. If Dagster added framework-level instrumentation with the OpenTelemetry API, then customers could install a vendor-specific SDK and immediately get telemetry, and then be able to do log/metric analysis, alerting, etc in the platform of their choice. Intro to OpenTelemetry with Python: https://www.youtube.com/watch?v=uIImfoLUi24 .