Multiple LocalExecutors on HA Scheduler deployment? #26564
Replies: 1 comment 1 reply
-
Yes. It should work this way. This was not the main purpose of multiple Scheduler implementation, but in principle it should work this way and those executors should work. I think one problem with this setup is the scenario when you are restarting the scheduler. All your taks will be also killed then and it will take a bit of time for the other schedulers to recognise them as zombies and restart/ fail them. So while this setup should be fine to run, you might have some edge cases. It does allow more througput but it's "high-availability" functionality is thus impaired. This is the main reason we do not explain it as a 'recommended" configuration. But if you are ready to bite the bullet, you can likely go this way. Also usually when you run single Local Executor, you can rely on the fact that the tasks share the same instance (And some people rely on this and exchnage data between tasks via temporary files etc.). This will not worka at all when you have multiple schedulers unless you also share the filesystem. Multiple Scheduler is mainly targeted for K8S and Celery executors - where tasks are executed independently from the scheduler and this way through "high-availability" can be achieved - for example when tasks are running in Kubernetes, and one of the scheduler stops, the others will adopt the running tasks (and the tasks will continue running) so the disruptions will be minimal (at most slight delay with updating task status. When you abruptly stop running schediuler wtth locl executor - not only your tasks will be killed but also it will take quite some time (multiple minutes) for it to recover. |
Beta Was this translation helpful? Give feedback.
-
According to Airflow documentation, it's now possible to run multiple Airflow Schedulers starting with Airflow 2.x.
What's not clear from the documentation is what happens if each of the machines running the Scheduler has
executor = LocalExecutor
in the[core]
section of airflow.cfg. In this context, if I have Airflow Scheduler running on 3 machines, does this mean that there will also be 3 LocalExecutors processing tasks in a distributed fashion?Beta Was this translation helpful? Give feedback.
All reactions