Apache Airflow version: 1.10.11
Environment: Service as part of a Docker Swarm
What happened:
Setting min_file_process_interval to a high value delays the execution of DAGs by up to the time specified for min_file_process_interval.
What you expected to happen:
We use Airflow as a deployed Docker container. We modify our DAGs locally and deploy a new version of the image every now and then. Thus the DAG definitions the deployed Airflow service uses will never update while the service runs, and to save processing resources, we expected setting min_file_process_interval will make Airflow very seldom look for updated DAG definitions.
From the documentation
after how much time (seconds) a new DAGs should be picked up from the filesystem
This sounds to me as the delay between checks for updated DAG definitions, so I don't understand why this setting delays DAG executions.
Apache Airflow version: 1.10.11
Environment: Service as part of a Docker Swarm
uname -a): Linux 5ed6acafbcee 5.3.0-1034-aws Adding support for catalog option to PrestoHook + refactor #36-Ubuntu SMP Tue Aug 18 08:58:43 UTC 2020 x86_64 GNU/LinuxWhat happened:
Setting
min_file_process_intervalto a high value delays the execution of DAGs by up to the time specified formin_file_process_interval.What you expected to happen:
We use Airflow as a deployed Docker container. We modify our DAGs locally and deploy a new version of the image every now and then. Thus the DAG definitions the deployed Airflow service uses will never update while the service runs, and to save processing resources, we expected setting
min_file_process_intervalwill make Airflow very seldom look for updated DAG definitions.From the documentation
This sounds to me as the delay between checks for updated DAG definitions, so I don't understand why this setting delays DAG executions.