Support disabling of dag_dir_list_interval

[arjunanan6]

Apache Airflow version

2.5.0

What happened

We have a unique situation with our Airflow deployment, in that we bake the DAGs into the image pre-deployment, meaning that a deployment is triggered every time there is a new DAG contribution.

During a recent performance troubleshooting, we have uncovered that the scheduler consumes much more memory than it should, and while looking for ways to optimize it, we came across the dag_dir_list_interval configuration. It's currently set to the default of 5 minutes. However, it seems like there is no possibility to disable it barring the possibility of setting it to a really high value like 86400.

What you think should happen instead

It would be nice if the parameter supports a value, say -1 that disables the scan altogether.

How to reproduce

Default configuration, the scan takes place every 5 minutes.

Operating System

Debian GNU/Linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow deployed with official helm chart on AKS.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hussein-awala]

happy to work on this

[potiuk]

Assigned you