[AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator #11119
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Both BaseOperator and XComArgs implement bit shift operators used to chain tasks in DAGs. By extracting this logic to new mixin we reduce code duplication and make it easier to implement it in the future. This change introduces also root property that allows us to reduce number of type checking. (cherry picked from 0779688)
boring-cyborg
bot
added
area:docs
area:serialization
area:webserver
|
We started using |
…ubDagOperator apache#10153 (cherry picked from commit 49c193f)
yuqian90
force-pushed
the
v1-10-test_with_task_group
branch
from
d4ef251
to
2a8e9a5
Compare
|
Hi @yuqian90 apologies but I need to close this as we're already feature freezed on 1.10. 2.0 Alpha is coming out in the next 1-2 weeks and at this point the only things we want to add to 1.10 are critical bugfixes and migration steps/scripts. Anyone who wants to use this feature will have full access to it when the alpha comes out :). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is cherry-picked from #10153. It makes
TaskGroupwork with v1-10-test so that we can encourage more people to start usingTaskGroup.(
TaskMixinwas also cherry-picked from #10930).Limitations:
www_rbac/views.pyandgraph.htmlworks withTaskGroup. So users will only be use TaskGroup in Graph View if they haverbac = Trueunder[webserver]inairflow.cfg. This is becausewww/views.pyand graph.html has too many differences against the master branch, making it hard to merge. IfTaskGroupis used in DAGs but the webserver is not rbac, the Graph View will not break, it'll just appear likeTaskGroupwas never used. All tasks will appear on the original DAG normally.This was the original commit message in #10153:
This PR introduces
TaskGroup, which is a simple UI task grouping concept.What this PR does:
>>and<<operatorsWhat this PR does not do:
SubDagOperator. AlthoughTaskGroupis intended as an alternative forSubDagOperator, deprecatingSubDagOperatorwill need to be discussed/implemented in the future.TaskGrouphandling in the Graph View. In places such as Tree View, it will look like as-ifTaskGroupdoes not exist and all tasks are in the same flat DAG.GitHub Issue: #8078
AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
This is the example_task_group that this PR adds:
