Running multiple celery workers on the same machine - is pidfile required?

[aran3]

I am running multiple celery worker on the same machine, each with different queues and different concurrency configuration.
This worked well in Airflow 1.10.15, but the same command now fails with airflow 2.1.2/2.1.3 because of ERROR: Pidfile ... already exists.
Is it expected that each worker will be started with the --pid parameter, even without running as daemon mode? Why has this not been the case in 1.10.15?

Full Error:

ERROR: Pidfile ${AIRFLOW_HOME}/airflow-worker.pid) already exists.
Seems we're already running? (pid: 91897)

Each worker is called with airflow celery worker -q 'queue_a,queue_b' -c 8 -H worker_name@%h and monitored externally. Worked on version 1.10.15, celery 4.4.7 where no airflow-worker.pid was created by any worker. Error on 2.1.2/2.1.3 with same celery 4.4.7 since each worker tries to create the file.

[potiuk]

If you want to run several airflow workers on single machine (which is not really common) you can by providing --pid parameter where you can provide a pid file where processs id should be stored. If you provide different pidfile for your different workers, it will work

[aran3]

So passing --pid is required for running multiple celery workers on the same machine?
Was this changed in 2.x, or was this just not working in 1.10?
BTW I could not find anything in the documentation about this - should probably be added if it is really a requirement in 2.x.

[potiuk]

It's not common to run several workers on the same machine (it's a really rare case). If it was working on 1.10 without it, that was probably a bug (and this is pretty common setup for any tool to have PID file and exclusive lock on it, so airflow just follows best practices on damonising and running as service.

BTW I could not find anything in the documentation about this - should probably be added if it is really a requirement in 2.x.

Absolutely - great idea! Airflow is community driven project - anyone can submit such documentation. So feel free to add such information in the place you'd look for it - I think you are the best person to know where to add it and use wording that would help people like you to find this useful. It's the best way to contribute back, thanking all the people who make the software free for use for you.

It's super easy - at the right page of the docs there is "suggest change on this page" button - and it will open a PR in GitHub and you will be able to open PR and contribute the doc change directly from within GitHub UI (without having dev environment even). Feel free to open such a change - you can ping me there - happy to review and approve it when you do it !