Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIP-31] Add dependency relationship on operator when templated_field is XComArg #8054

Closed
casassg opened this issue Apr 2, 2020 · 0 comments · Fixed by #8805
Closed

[AIP-31] Add dependency relationship on operator when templated_field is XComArg #8054

casassg opened this issue Apr 2, 2020 · 0 comments · Fixed by #8805
Assignees
@casassg
Labels
AIP-31 Task Flow API for nicer DAG definition kind:feature Feature Requests

Comments

@casassg
Copy link
Contributor

casassg commented Apr 2, 2020

Description

XComArg allow us to set explicit XCom messages between operators. Adding a dependency between Operators is needed as it does not make sense to execute an operator that requires an XCom from another operator.

Use case / motivation

As a DAG Author, I want Airflow to automatically detect that an operator that I initialize using an XComArg is set as a downstream dependency from the XCom origin operator.

Related Issues

Blocked by: #8052
@casassg casassg added the kind:feature Feature Requests label Apr 2, 2020
@turbaszek turbaszek added the AIP-31 Task Flow API for nicer DAG definition label Apr 2, 2020
@casassg casassg mentioned this issue Apr 25, 2020
Closed
6 tasks
turbaszek added a commit to databand-ai/airflow that referenced this issue May 5, 2020
@turbaszek
Resolve upstream tasks when template field is XComArg
529e69b 
closes: apache#8054
@turbaszek turbaszek mentioned this issue May 5, 2020
Closed
6 tasks
turbaszek added a commit to databand-ai/airflow that referenced this issue May 10, 2020
@turbaszek
Resolve upstream tasks when template field is XComArg
d40c9d9 
closes: apache#8054
@turbaszek turbaszek mentioned this issue May 10, 2020
Merged
6 tasks
turbaszek added a commit to databand-ai/airflow that referenced this issue Jun 9, 2020
@turbaszek
Resolve upstream tasks when template field is XComArg
daf4299 
closes: apache#8054
turbaszek added a commit that referenced this issue Jun 15, 2020
@turbaszek @ashb
Resolve upstream tasks when template field is XComArg (#8805)
431ea32 
* Resolve upstream tasks when template field is XComArg

closes: #8054

* fixup! Resolve upstream tasks when template field is XComArg

* Resolve task relations in DagRun and DagBag

* Add tests for serialized DAG

* Set dependencies only in bag_dag, refactor tests

* Traverse template_fields attribute

* Use provide_test_dag_bag in all tests

* fixup! Use provide_test_dag_bag in all tests

* Use metaclass + setattr

* Add prepare_for_execution method

* Check signature of __init__ not class

* Apply suggestions from code review

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

* Update airflow/models/baseoperator.py

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
kaxil pushed a commit to kaxil/airflow that referenced this issue Jun 27, 2020
@turbaszek @ashb @kaxil
Resolve upstream tasks when template field is XComArg (apache#8805)
3e8a3e1 
* Resolve upstream tasks when template field is XComArg

closes: apache#8054

* fixup! Resolve upstream tasks when template field is XComArg

* Resolve task relations in DagRun and DagBag

* Add tests for serialized DAG

* Set dependencies only in bag_dag, refactor tests

* Traverse template_fields attribute

* Use provide_test_dag_bag in all tests

* fixup! Use provide_test_dag_bag in all tests

* Use metaclass + setattr

* Add prepare_for_execution method

* Check signature of __init__ not class

* Apply suggestions from code review

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

* Update airflow/models/baseoperator.py

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
@alex-astronomer alex-astronomer mentioned this issue Oct 1, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@casassg casassg

Labels
AIP-31 Task Flow API for nicer DAG definition kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Resolve upstream tasks when template field is XComArg databand-ai/airflow
2 participants
@casassg @turbaszek