Skip to content

use AIRFLOW__CELERY__POOL=solo when debugging celery-worker#56868

Merged
potiuk merged 2 commits intoapache:mainfrom
rawwar:kalyan/breeze/debug/change_celery_default_pool
Oct 21, 2025
Merged

use AIRFLOW__CELERY__POOL=solo when debugging celery-worker#56868
potiuk merged 2 commits intoapache:mainfrom
rawwar:kalyan/breeze/debug/change_celery_default_pool

Conversation

@rawwar
Copy link
Contributor

@rawwar rawwar commented Oct 20, 2025

by default, we use pre-fork and Its not possible to debug these as each of them run in separate process. Changing it to solo as this makes tasks to run in main thread in sequential order. Should be helpful when debugging. I'm trying to make this work. But, as of now, #56865 is blocking me to continue.

@boring-cyborg boring-cyborg bot added area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Oct 20, 2025
@rawwar
Copy link
Contributor Author

rawwar commented Oct 20, 2025

Edit: It is possible to debug. But, as multiple tasks are going to be run in parallel, its very hard to debug. IDE jumps between processes and its confusing. Having pool set to Solo made my experience much easier

@rawwar rawwar marked this pull request as ready for review October 21, 2025 00:36
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@potiuk potiuk merged commit c99f814 into apache:main Oct 21, 2025
112 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 21, 2025
…er (#56868)

* use AIRFLOW__CELERY__POOL=solo when debugging celery-worker

* Update scripts/in_container/bin/run_tmux
(cherry picked from commit c99f814)

Co-authored-by: Kalyan R <kalyan.ben10@live.com>
@github-actions
Copy link

Backport successfully created: v3-1-test

Status Branch Result
v3-1-test PR Link

@rawwar rawwar deleted the kalyan/breeze/debug/change_celery_default_pool branch October 21, 2025 08:28
@rawwar rawwar removed the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Oct 21, 2025
Jonpaco23 pushed a commit to Jonpaco23/airflow that referenced this pull request Oct 21, 2025
…6868)

* use AIRFLOW__CELERY__POOL=solo when debugging celery-worker

* Update scripts/in_container/bin/run_tmux
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants