-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Add example of TaskFlow to declare task dependencies #17927
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
As a new user of Airflow it is not obvious to realize TaskFlow exists and one might think declaring dependencies between tasks is very verbose. Having more examples of such would help in promoting this style of writing DAGs.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
|
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
|
@ee987, sorry, wanted to get this merged but there's now a static check failure. Please can you address it, thanks |
| third_task.set_upstream(second_task) | ||
|
|
||
| These both do exactly the same thing, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. | ||
| Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`:: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`:: | |
| Option 3: The ``@task`` decorator provided by TaskFlow API, see :ref:`TaskFlow <concepts:taskflow>`:: |
| @task | ||
| def second_task(some_parameter): | ||
| ... | ||
| @task | ||
| def first_task(): | ||
| ... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| @task | |
| def second_task(some_parameter): | |
| ... | |
| @task | |
| def first_task(): | |
| ... | |
| @task | |
| def second_task(some_parameter): | |
| ... | |
| @task | |
| def first_task(): | |
| ... |
(I hope the linter won't complain this needs two empty lines.)
| second_task(first_task()) | ||
|
|
||
|
|
||
| The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the @task-decorator, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`. | |
| The former two do exactly the same thing and work on any type of Operator, but in general we recommend you use the bitshift operators, as they are easier to read in most cases. The TaskFlow API likewise connects dependencies between tasks, but is tailored to python functions written with the ``@task-decorator``, note that the functions here are not executed inline, rather they are converted to tasks to be scheduled and sent to Executors, for more information see :ref:`TaskFlow <concepts:taskflow>`. |
Also the CI says ``concepts:taskflow` is not a defined reference. Where do you intend to link this to? (There's another occurrence of this above.)
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
As a new user of Airflow it is not obvious to realize TaskFlow exists and
one might think declaring dependencies between tasks is very verbose.
Having more examples of such would help in promoting this style of
writing DAGs and speed up on-boarding.