Should task survive after the DAG is timeout? #16901
Unanswered
Shadowsong27
asked this question in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
This discussion is related to #14321
I recently upgraded my Airflow to 2 and start to realise this issue (I thought it's a feature) has been resolved.
In my case, I actually would like task to survive and to finish even if the DAG timeout. Consider the following scenarios:
Scenario 1:
for certain long running process like model training, if for some reason the process took longer than expected, it might exceed the timeout duration, failing or skipping the task would kill the process and wasted the time spent earlier.
Scenario 2:
during debugging, we need to frequently clear partial DAGs, like tasks in different branches, which will keep the DAG in
running
state for a long time, eventually we might end with having a task just triggered and set toSKIP
orFAILED
externally because this is a new task added and scheduled.For potential solution or suggestions,
micro-batch
processing, but it seems relevant for longer and bulkier tasksBeta Was this translation helpful? Give feedback.
All reactions