Airflow deployed on Kubernetes cluster NOT showing airflow app metrics in STATSD Exporter

[dibyo-gintaa]

Apache Airflow version

2.1.1

Operating System

PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian

Versions of Apache Airflow Providers

airflow:
extraPipPackages:

• apache-airflow[statsd]

Deployment

Official Apache Airflow Helm Chart

Deployment details

We have deployed the airflow as a helm release in a Kubernetes cluster.
So airflow web server is running as a pod in kubernetes cluster having statsd metrics in its values.yaml.

We have tried to follow this link: https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/metrics.html

We also deployed statsd exporter as a helm release.
Our flow is like this: airflow app--> statsd config in values.yaml --> running statsd exporter where it will receive airflow metrics --> send the metrics to Prometheus server.
We have given the follow config in airflow values.yaml:
[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

What happened

We can not see any airflow metrics running in Host: 8125 or Host:125
When we do a local port forward of the statsd exporter pod we see only default statsd metric but not any airflow related metrics in statsd

What you expected to happen

No response

How to reproduce

1. Deploy airflow cluster as helm release using values.yaml
   helm install poc-cluster-airflow airflow-stable/airflow --namespace airflow --version 8.4.1 --values values-custom.yaml
   values-custom.yaml having:

Config settings to go into the mounted airflow.cfg

config:
metrics:
#statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_on: True
statsd_port: 8125
statsd_prefix: airflow
statsd_host: '{{ printf "%s-statsd" .Release.Name }}'

2. helm --kubeconfig kube_config_cluster.yml install statsd-release prometheus-community/prometheus-statsd-exporter --namespace airflow

3. Try to port forward the statsd exporter and see metrics in localhost:9102/metrics.
   No airflow metrics is coming up

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!