Airflow tutorial to use functional DAGs #11308
Conversation
|
The Build Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
| # [START documentation] | ||
| dag.doc_md = __doc__ | ||
| # [END documentation] |
There was a problem hiding this comment.
Should we move this to DAG invocation?
| data_string = u'{"1001": 301.27, "1002": 433.21, "1003": 502.22}' | ||
| return data_string | ||
| # [END extract] | ||
| extract.doc_md = """\ |
There was a problem hiding this comment.
If I'm not mistaken we should be able to do:
@task(doc_md="here goes the docs")
def extract(**kwargs):
...
|
@vikramkoka I think this a good example 👍 |
|
Looks good. I would try to use multiple outputs better instead of json dumping and such. That feels against what we have been trying to do. Lets return dictionary and load the number directly |
|
A thought: should add somewhere the |
Thank you both for the detailed, prompt feedback! I really appreciate it and will make the changes. |
|
(note that you may need to clean up code, my code suggestions was mostly to point how to use it) Also, may be interesting to add a unit test that executes the DAG end to end? |
Yes, absolutely. Will make the changes and keep updating the PR, just so that you have the visibility. |
Your point is well taken and I should have stated up front, that I would like to add one more example which is more complex which contains both the 'get_current_context', as well as a BashOperator to demonstrate how to use functional Dags with non-python (more classic) operators. |
Plus +1 for that, this should be simple as adding this DAG id to: airflow/tests/test_example_dags_system.py Lines 24 to 31 in b0fcf67 |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
docs/tutorial_functional.rst
Outdated
|
|
||
| What's Next? | ||
| ------------- | ||
| You have seen how simple it is to write DAGs using the Functional DAG paradigm within |
There was a problem hiding this comment.
Would be nice to have this point to https://github.com/apache/airflow/blob/master/docs/concepts.rst#functional-dags and https://github.com/apache/airflow/blob/master/docs/concepts.rst#functional-dags more specifically.
Also, it may be interesting to link the AIP-31 for more insight into this and potentially my talk and Jonathan's article as learn more resources? CC @turbaszek for thoughts
There was a problem hiding this comment.
+1 for linking AIP and this article for more insight:
https://medium.com/databand-ai/aip-31-airflow-functional-dag-definition-b34852a632d0
There was a problem hiding this comment.
I added the links to the AIP and to the Functional DAGs section of the Concept doc.
I was not sure if it was ok to add a link to an external site which has a paywall (Medium) from the Apache open source docs, so did not add that.
Created a new Airflow tutorial to use Decorated Flows (a.k.a. functional DAGs). Also created a DAG to perform the same operations without using functional DAGs to be compatible with Airflow 1.10.x and to show the difference.
It makes sense to simplify the return variables being passed around without needlessly converting to JSON and then reconverting back. Co-authored-by: Gerard Casas Saez <casassg@users.noreply.github.com>
Fixed data passing between tasks to be more natural without converting to JSON and converting back to variables.
Based on feedback on the PR, updated the DAG options (including schedule) and the fixed the task documentation to avoid indentation.
Added the tutorial documentation to the docs directory. Fixed linting errors in the example dags. Tweaked some doc references in the example dags for inclusion into the tutorial documentation. Added the example dags to example tests.
Had a multiple_outputs=True defined in the Extract task defn, which was unnecessary. - Removed based on feedback.
Added word summarization to the spelling list in the docs directory to fix spell check error in tutorial. Also fixed another sentence error in the tutorial document.
|
Ran tests local (this time on the right commit) so merging. |
Created a new Airflow ETL tutorial to use functional DAGs. Also created a DAG to perform the same operations without using functional DAGs to be compatible with Airflow 1.10.x and to show the difference between the "functional" and the "classic" ways side by side to illustrate the differences.
I tried to make this simple and not rely on any operators. I will create the html documentation page to cover this as soon as I get confirmation from @casassg and @turbaszek that this is in the right general direction. Also tagging @ashb for general review.
closes: #9041
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.