New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CeleryExecutor adopts any task with an external_executor_id #15457
Comments
@leonsmith are you interested in fixing this? We can target the next release with the fix |
I don't mind having a go, it is pretty low on my list though as we worked around it by cancelling running tasks when switching executors. |
I was not able to reproduce this in main, I suspect that it has been resolved. Can you try it on main or 2.1.0 @leonsmith |
I changed the external_executor_id as you said and made sure that the task was queued before changing it. It adopted the tasks and ran them. It wasn't stuck in queued |
It seems that part of the issue here, to be specific the following part:
will be fixed by this PR #16550 |
Closed by #16550 - This was already released in Airflow 2.1.1 |
Apache Airflow version: 2.0.1
Kubernetes version (if you are using kubernetes) (use
kubectl version
): N/AEnvironment:
uname -a
): Linux 4.14.225-168.357.amzn2.x86_64What happened:
Switching to a
CeleryExecutor
from another executor (that makes use ofexternal_executor_id
) results in Celery adopting the task and the task staying in the queued state indefinitely.What you expected to happen:
Celery to correctly return
TaskInstances
it can't adopt, allowing them to be re-submitted by airflow.How to reproduce it:
Queue a
TaskInstance
in some manner but make sure it stays in the queued state and doesn't progress to running.(You can do this via using the
CeleryExecutor
but not starting a worker)Kill the Executor and adjust the
external_executor_id
of the queuedTaskInstance
to some random string(To simulate a task being queued by a different executor of which Celery knows nothing about)
Start the
CeleryExecutor
& corresponding workers.Observe that the task gets "adopted" by the
CeleryExecutor
as it passes the following snippetObserve the task instance stays in the queued state indefinitely as the return of the following code is an empty mapping
Anything else we need to know:
N/A
The text was updated successfully, but these errors were encountered: