Airflow can be set up to send metrics to StatsD.
First you must install statsd requirement:
pip install 'apache-airflow[statsd]'Add the following lines to your configuration file e.g. airflow.cfg
[scheduler]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow| Name | Description |
|---|---|
| <job_name>_start | Number of started <job_name> job, ex. SchedulerJob, LocalTaskJob |
| <job_name>_end | Number of ended <job_name> job, ex. SchedulerJob, LocalTaskJob |
| operator_failures_<operator_name> | Operator <operator_name> failures |
| operator_successes_<operator_name> | Operator <operator_name> successes |
| ti_failures | Overall task instances failures |
| ti_successes | Overall task instances successes |
| zombies_killed | Zombie tasks killed |
| scheduler_heartbeat | Scheduler heartbeats |
| Name | Description |
|---|---|
| collect_dags | Seconds taken to scan and import DAGs |
| dagbag_import_errors | DAG import errors |
| dagbag_size | DAG bag size |
| dag_processing.last_runtime.<dag_file> | Seconds spent processing <dag_file> (in most recent iteration) |
| dag_processing.last_run.seconds_ago.<dag_file> | Seconds since <dag_file> was last processed |
| executor.open_slots | Number of open slots on executor |
| executor.queued_tasks | Number of queued tasks on executor |
| executor.running_tasks | Number of running tasks on executor |
| pool.open_slots.<pool_name> | Number of open slots in the pool |
| pool.used_slots.<pool_name> | Number of used slots in the pool |
| pool.starving_tasks.<pool_name> | Number of starving tasks in the pool |
| Name | Description |
|---|---|
| dagrun.dependency-check.<dag_id> | Seconds taken to check DAG dependencies |
| dag.<dag_id>.<task_id>.duration | Seconds taken to finish a task |
| dag.loading-duration.<dag_id> | Seconds taken to load the given DAG |
| dagrun.duration.success.<dag_id> | Seconds taken for a DagRun to reach success state |
| dagrun.duration.failed.<dag_id> | Seconds taken for a DagRun to reach failed state |
| dagrun.schedule_delay.<dag_id> | Seconds of delay between the scheduled DagRun start date and the actual DagRun start date |