Different Parallelism for different workers

[ErandM50]

I would appreciate if someone could give some insights on the questions below. I am trying to understand how to set different parallelism for different workers in a cluster.

• When we have a cluster with multiple workers and CeleryExecutor, is it possible to set different parallelism for each of the workers? For instance, if in Worker A I set worker parallelism (AIRFLOW__CELERY__WORKER_CONCURRENCY) to 32 in the cfg file and in Worker B I set to 64, would that work?

• In the case of LocalExecutor (say we run scheduler on 2 workers), how can this be done? Do I set different parallelism (AIRFLOW__CORE__PARALLELISM) on each, will that work?

[potiuk]

• When we have a cluster with multiple workers and CeleryExecutor, is it possible to set different parallelism for each of the workers? For instance, if in Worker A I set worker parallelism (AIRFLOW__CELERY__WORKER_CONCURRENCY) to 32 in the cfg file and in Worker B I set to 64, would that work?

Yes. See https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#worker-concurrency

• In the case of LocalExecutor (say we run scheduler on 2 workers), how can this be done? Do I set different parallelism (AIRFLOW__CORE__PARALLELISM) on each, will that work?

Not sure. But it's easy - to check. Just run them and see.