-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure teardown doesn't run until all other tasks complete #29332
Comments
Hi all, I'm encountering an issue with the setup and teardown feature in Airflow, which I believe is related to this issue. When executing a DAG with the provided setup script, the teardown task does not wait for dependent tasks to finish before executing. Specifically, if I clear task t1 while task t3 has any of the following statuses: finished, failed, or upstream_failed, the teardown task executes immediately without waiting for t1 to complete. Steps to Reproduce: Execute the provided DAG setup script. Actual Behavior: Setup Script:
Additional Information: Airflow Version: 2.7.1 Video: airflow-setup-teardown-error.movIs this an expected behavior for this feature at the moment? It seems problematic, specially when using dbt with cosmos, and we end up having the teardown of our cluster while some tasks are still running. |
Can you please double-check the documentation in https://airflow.apache.org/docs/apache-airflow/stable/howto/setup-and-teardown.html#setup-and-teardown ? The setup task in your example is not marked as setup. |
Thank you @jscheffl, I modified the example with this other syntaxe:
But I see the same behavior. Which is expected since the syntaxes are equivalent 🤔 Could this actually be a bug? |
Okay, I did a regression and can confirm this bug. Note that the additional dependency from Setup >> Teardown is not needed. It is modelled automatically.
The error exactly appears if you clear a normal task between setup and teardown which is NOT the last task before teardown and select no downstream tasks. Means that the setup is executed, then the real cleared task. The bug is that the only direct pre-ceeding task before teardown is already finished such that for the scheduler it seems to be "ready for teardown". Scheduler does not check whether all tasks between setup and teardown are completed. In the example above the error is generated if either task "normal" or "normal2" are cleared w/o downstream. It works as expected if "normal3" is cleared. |
looks like what we need to do is find all the tasks “in the scope” of the teardown (and that is, those tasks that are in between the teardown and its setup(s)) and then just make sure they are not running. one idea for fixing: but, it would probably be more efficient if we did not hit that block unless the direct upstreams already passed so another option would be to instead add something like this to the end then it would only evaluate the indirect in-scope tasks are done if everything else passed. (i'm missing an else in that screenshot to guarantee it's a teardown) ideally perhaps, we would also be able to know that this run was the result of a clear and then don’t bother to run this check generally speaking! |
No description provided.
The text was updated successfully, but these errors were encountered: