Airflow Metrics Page Misses Some Metrics

[rafidka]

Apache Airflow version:

1.10.11rc2

Kubernetes version (if you are using kubernetes) (use kubectl version):

N/A

Environment:

N/A

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

By inspecting the code of Airflow (1.10.11rc2, though I believe previous versions apply as well), I noticed the metrics documentation page misses the following metrics:

• {job_name}_heartbeat_failure
• dag_file_refresh_error
• scheduler.tasks.executable
• scheduler.tasks.pending
• scheduler.tasks.running
• scheduler.tasks.starving
• scheduler.tasks.without_dagrun
• task_removed_from_dag.{dag_id}
• task_restored_to_dag.{dag_id}
• task_instance_created-{operator_name}

Any reason why these are not documented?

Additionally, the following metrics are incorrectly documented:

• dag_processing.processor_timeouts: documented as a guage but it is actually a counter
• dag_processing.last_runtime.{dag_filename}: documented as a gauge but it is actually a timer

What you expected to happen:

• The Metrics page should document all metrics being emitted by Airflow.
• The Metrics page should correctly document the type of the metric.

How to reproduce it:

• Clone the latest Airflow 1.x code base.
• Search the code base for Stats., i.e. lines that publish metrics using the Stats object.
• Compare the result with Metrics page in Airflow documentation.

Anything else we need to know:

I am happy to send a PR to update the documentation.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

Does this problem also occur on the master branch?
https://airflow.readthedocs.io/en/latest/metrics.html

[rafidka]

Yes, I just checked and apart from a couple of the metrics in my list above which seem to be documented, the rest are still undocumented. Also, the two metrics which are wrongly documented as gauges are still so in the page you linked.

[rafidka]

Any update on this?

[mik-laj]

@rafidka Can you create a PR? This way, it is easier to discuss specific changes. I only know now that not all documentation changes may be available in Airflow 1.10 and this is still a problem, but a less important one. We are now trying to focus on developing the documentation for the master branch as we want to release Airflow 2.0 soon. If there is a problem, I think it is worth making the necessary improvements.

[rafidka]

@mik-laj sure, I will do.