Metrics - Introducing a counter for number of DAGs in "running" state

[SolbiatiAlessandro]

Description

Could we introduce a new counter in Airflow Metrics to track number of DAGs that are in 'running' state?

I am aware of the existing dag_processing.processes counter that is documented as Number of currently running DAG parsing processes. This counter track how many dags are being parsed, not how many are runnings.

I am aware of the existing executor.queued_tasks and executor.running_tasks, but they are task-wise and not dag-wise.

Use case / motivation

Consider this example

I want to know how many DAGs are in the state of running to understand memory consumption issues and to understand how long is my "queue" of DAGs. To my understanding there is no metric currently that I can use to track that.

I propose a counter executor.running_dags that would return the number of DAGs in the running state.

Does this make sense? Am I missing something? If folks I agree I would be happy to work on a PR :)

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

This information is in the database so you can use:
https://github.com/robinhood/airflow-prometheus-exporter or https://github.com/PBWebMedia/airflow-prometheus-exporter or other similar exporter

We use statsd metrics only for information that we do not have in the database, i.e. runtime metrics.

[SolbiatiAlessandro]

@mik-laj thanks for the repsonse! I will look into that :)
Do you think we should add two lines in this doc page to mention that statsd only track non-database information and you can find more metrics in the database? https://airflow.apache.org/docs/stable/metrics.html

[mik-laj]

@SolbiatiAlessandro Yes. Please do it. ;-)

[ashb]

Additionally: we can't (easily) report this as a metric now that running more than on e scheduler is supported.

[dskoda1]

Hi, is there any chance getting these metrics published via the default statsd set can be re-visted? Being able to tell how many DAGs are currently queued (or other states), and which ones, is an incredibly useful metric. It seems a bit round about to not have this as part of core airflow.

We would be open to contributing this since it would be so useful to have, specifically so we don't need to bring in a third party prometheus exporter.

[potiuk]

We would be open to contributing this since it would be so useful to have, specifically so we don't need to bring in a third party prometheus exporter.

Sure. Go ahead.

[Rivgad]

Is there any chance this will be implemented? It would be really useful metric since SLA don't work properly

[potiuk]

Is there any chance this will be implemented? It would be really useful metric since SLA don't work properly

Surely. If someone (for example you ?) implement it. This is the best way of making sure it is implemented. I also marked it as "good-first-issue" and if not you, then hopefully someone will pick it up.

This is how open-source development works. Things get implemented when somoene picks an interest in things and implements them.

[Rivgad]

@potiuk thanks for your reply! I think i would like to implement this. At least i'll try to contribute