How to define custom metrics

[aeroyorch]

Is there any recommended way to define and emit custom metrics in Airflow?

For example using custom stats like:

from airflow.sdk.observability import stats

stats.gauge("my_service.queue_depth", 42)
# ...and similarly stats.incr(...) / stats.decr(...) / stats.timing(...) / stats.timer(...)

If this is the case, a dedicated "Custom Metrics" section in airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst would be very helpful.

[apoorva-01]

Yeah, that's basically the supported way. In Airflow 3 the metrics client moved to the task SDK, and the import you wrote is the right one:

from airflow.sdk.observability import stats

stats.gauge("my_service.queue_depth", 42)
stats.incr("my_service.processed")
stats.decr("my_service.in_flight")
stats.timing("my_service.batch_ms", 1234)      # ms or a timedelta
with stats.timer("my_service.batch"):          # times the block
    ...

from airflow.stats import Stats still works but it's deprecated now (it just warns and forwards to the same place), so the airflow.sdk.observability path is the one to use going forward.

The one gotcha that trips people up: these calls are no-ops unless you actually have a metrics backend turned on. Out of the box nothing is emitting, so your gauge goes nowhere and it looks broken. Turn one on in config:

[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

or the OpenTelemetry equivalent with otel_on = True.

Also watch [metrics] metrics_allow_list / metrics_block_list if you've set them. Allow-list is prefix-matched, so if it's configured and your my_service.* prefix isn't in it, the custom metric gets filtered out silently (and if you set an allow-list, the block-list is ignored entirely). Signatures if you need the rest: incr/decr(stat, count=None, rate=None, *, tags=...), gauge(stat, value, *, tags=...), all take a tags dict for dimensional metrics.

Agree a dedicated "Custom Metrics" section in metrics.rst would help, this comes up a lot.

[aeroyorch]

Thanks! I mainly wanted to confirm that my understanding was correct. I don't have any problem adding these updates to the metrics docs :).