-
Notifications
You must be signed in to change notification settings - Fork 14k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Execute SubDAG tasks as part of parent DAG #8078
Comments
+1 |
I think it's totally reasonable to say that SubDag separation should be more of a UI feature than an execution feature, so I think if we can identify which tasks are "subdag" tasks, we can condense them in the UI. |
@kaxil does this mess with DAG serialization? Would this subdag task adding happen at DAG parsing time? |
Yeah, totally agree. I don't think the added complexity with nested dags gives more values than errors. I actually have a more simplistic thought. Instead of returning a DAG from the |
I like the idea.
No that should work fine |
@kaxil @dimberman Do we need an AIP for this? This seems to change the whole behavior of the |
Yes, please. This definitely needs voting and some design discussion. Can you please create an AIP and start a discussion thread once you have that AIP doc ready. |
Sounds good. I will draft an AIP within this week and open a thread for discussion. |
@xinbinhuang SGTM! I'm gonna play around with a Draft PR just to see what I can jerry-rig, but it will be more based around "what CAN we do" than "what WILL we do" |
Done |
@kaxil @turbaszek @dimberman I found this PR #5498 when I was working on the AIP draft. It fixed the It seems to me that the problem has been fixed? I wonder if it is still useful for the suggested change and proceed to finish the AIP draft? |
I think I will proceed with finishing the AIP and then let the community decide. I think it makes more sense to treat |
Yes, please do 👍 |
…ubDagOperator (#10153) This commit introduces TaskGroup, which is a simple UI task grouping concept. - TaskGroups can be collapsed/expanded in Graph View when clicked - TaskGroups can be nested - TaskGroups can be put upstream/downstream of tasks or other TaskGroups with >> and << operators - Search box, hovering, focusing in Graph View treats TaskGroup properly. E.g. searching for tasks also highlights TaskGroup that contains matching task_id. When TaskGroup is expanded/collapsed, the affected TaskGroup is put in focus and moved to the centre of the graph. What this commit does not do: - This commit does not change or remove SubDagOperator. Although TaskGroup is intended as an alternative for SubDagOperator, deprecating SubDagOperator will need to be discussed/implemented in the future. - This PR only implemented TaskGroup handling in the Graph View. In places such as Tree View, it will look like as-if - TaskGroup does not exist and all tasks are in the same flat DAG. GitHub Issue: #8078 AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
Closing in favour of #10153 |
Description
Currently, the SubDagOperator launches a completely different DAG and then monitors it as a separate entity. This has lead to all sorts of edge case (e.g. when workers have different executors than the scheduler).
This Issue suggests that we instead merge all tasks from the subdag into the original DAG with an extra "original_dag" label. this means at the UI level we can still condense all subdag tasks into a single task, but a single scheduler will still perform all task launching.
Use case / motivation
We want SubDag tasks to act exactly the same as non-subdag tasks
Related Issues
The text was updated successfully, but these errors were encountered: