~airflow.executors.local_executor.LocalExecutor
runs tasks by spawning processes in a controlled fashion in different modes.
Given that BaseExecutor has the option to receive a parallelism
parameter to limit the number of process spawned, when this parameter is 0
the number of processes that LocalExecutor can spawn is unlimited.
The following strategies are implemented:
- Unlimited Parallelism (
self.parallelism == 0
): In this strategy, LocalExecutor will
spawn a process every timeexecute_async
is called, that is, every task submitted to the
~airflow.executors.local_executor.LocalExecutor
will be executed in its own process. Once the task is executed and the
result stored in theresult_queue
, the process terminates. There is no need for a
task_queue
in this approach, since as soon as a task is received a new process will be
allocated to the task. Processes used in this strategy are of class~airflow.executors.local_executor.LocalWorker
. - Limited Parallelism (
self.parallelism > 0
): In this strategy, the~airflow.executors.local_executor.LocalExecutor
spawns
the number of processes equal to the value ofself.parallelism
atstart
time,
using atask_queue
to coordinate the ingestion of tasks and the work distribution among
the workers, which will take a task as soon as they are ready. During the lifecycle of
the LocalExecutor, the worker processes are running waiting for tasks, once the
LocalExecutor receives the call to shutdown the executor a poison token is sent to the
workers to terminate them. Processes used in this strategy are of class~airflow.executors.local_executor.QueuedLocalWorker
.
Arguably, ~airflow.executors.sequential_executor.SequentialExecutor
could be thought as a LocalExecutor
with limited parallelism of just 1 worker, i.e. self.parallelism = 1
. This option could lead to the unification of the executor implementations, running locally, into just one ~airflow.executors.local_executor.LocalExecutor
with multiple modes.