Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
Hello,
We are using airflow 2.4.6 version
We are creating dags in Dags folder using file operations ( We have template created for DAG and new DAG is created using this template with some changes given by user)
So whenever user creates a files it gets saved in DAGs folder - which is common for all 3 components - Airflow Webserver, Scheduler and Celery instances
Issue : When user creates a file - it takes minutes to show up in webserver, in fact we can not run that DAG using API as it is not available.
We tried below settings in airflow.cfg
_dagbag_import_timeout = 120.0
dag_file_processor_timeout = 300
min_file_process_interval = 30
Howewq often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
dag_dir_list_interval = 30_
As mentioned above - we have set list interval as 30 seconds - and was expecting that after every 30 seconds , if there are new files then those will get processed and will be available in Webserver UI and for running the DAG as well
Can anyone help us out here as this is kind of critical blocker for us.
Thanks in Advance!
MB
What you think should happen instead
No response
How to reproduce
We are creating dags in Dags folder using file operations ( We have template created for DAG and new DAG is created using this template with some changes given by user)
So whenever user creates a files it gets saved in DAGs folder - which is common for all 3 components - Airflow Webserver, Scheduler and Celery instances
Issue : When user creates a file - it takes minutes to show up in webserver, in fact we can not run that DAG using API as it is not available. Sometime it takes upto 30 mins to show up in UI
Operating System
linux
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
Hello,
We are using airflow 2.4.6 version
We are creating dags in Dags folder using file operations ( We have template created for DAG and new DAG is created using this template with some changes given by user)
So whenever user creates a files it gets saved in DAGs folder - which is common for all 3 components - Airflow Webserver, Scheduler and Celery instances
Issue : When user creates a file - it takes minutes to show up in webserver, in fact we can not run that DAG using API as it is not available.
We tried below settings in airflow.cfg
_dagbag_import_timeout = 120.0
dag_file_processor_timeout = 300
min_file_process_interval = 30
Howewq often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
dag_dir_list_interval = 30_
As mentioned above - we have set list interval as 30 seconds - and was expecting that after every 30 seconds , if there are new files then those will get processed and will be available in Webserver UI and for running the DAG as well
Can anyone help us out here as this is kind of critical blocker for us.
Thanks in Advance!
MB
What you think should happen instead
No response
How to reproduce
We are creating dags in Dags folder using file operations ( We have template created for DAG and new DAG is created using this template with some changes given by user)
So whenever user creates a files it gets saved in DAGs folder - which is common for all 3 components - Airflow Webserver, Scheduler and Celery instances
Issue : When user creates a file - it takes minutes to show up in webserver, in fact we can not run that DAG using API as it is not available. Sometime it takes upto 30 mins to show up in UI
Operating System
linux
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct