Repository navigation
Adding (disabled) DAGs impacts schedule_delay metric? #14053
|
I've noticed my avg schedule_delay increases when I add a lot (~300) of disabled DAGs that have a schedule interval. Is this behaviour correct? I tried looking in the source code where this is implemented but couldn't find it. Thanks in advance, |
Replies: 1 comment 3 replies
|
I suspect you are on version 1.10.x, correct? In 1.10, the scheduler would only parse each dag after it had parsed the file (even though the parsing happened in a sub-process) -- this means that having a lot of extra files to process would slow down the scheduling. This was one of the big changes we introduced in 2.0.0 to break this link between parsing and scheduling, meaning that having extra files doesn't slow down the scheduling of active dags. Try 2.0.0 😁 |
|
Ah understood. Thanks for the explanation! And yes, looking forward to moving to 2.0.0 :) |
|
@ashb Also, could you point me to the file that does the scheduling? Both in 1.10.x and 2.0.0? Thanks in advance. |
|
It's split across two files but https://github.com/apache/airflow/blob/2.0.1/airflow/jobs/scheduler_job.py (Same file names on 1.10) |
I suspect you are on version 1.10.x, correct?
In 1.10, the scheduler would only parse each dag after it had parsed the file (even though the parsing happened in a sub-process) -- this means that having a lot of extra files to process would slow down the scheduling.
This was one of the big changes we introduced in 2.0.0 to break this link between parsing and scheduling, meaning that having extra files doesn't slow down the scheduling of active dags.
Try 2.0.0 😁