In Celery mode of airflow, does each worker only execute one task at a time?

[hrbust-liu]

I have a question. In Celery mode of airflow, does each worker only execute one task at a time?
I want to change a global variable inside a task. Will it be read by other tasks running in parallel in dag? (I expect it will not be read. I understand it should be an independent process environment)

Additional explanation: PythonOperator configures python_callable inside

[dominikhei]

One worker can also execute tasks concurrently, you can define this via AIRFLOW__CELERY__WORKER_CONCURRENCY and AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG.

What type of variable are you referring to? Airflow variables airflow.model.Variable ? Or are you referring to the global keyword in python?

[hrbust-liu]

Global variables of Python

[hrbust-liu]

I understand that AIRFLOW__CELERY__WORKER_CONCURRENCY represents the number of workers that can be executed at the same time. This may indicate the number of tasks/Operators that can be run at the same time
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG represents how many tasks of a dag can run in parallel.

It seems to be different from what I want to say. What I want is that, for example, if a dag is A, B, C, D, then A/B/C/D may run at the same time. Will they be on the same process, causing conflicts when updating global variables?

[potiuk]

Depends on your celery setting. By default Celery creates separate fork for each worker process - so they are separate interpreters, but you can run celery in the mode that they are sharing interpreter (solo mode). Up to you. Look it up in celery docs

[potiuk]

https://docs.celeryq.dev/en/stable/userguide/workers.html