Docs: doc_md + task docstrings for measurement correction example Dags#66707
Conversation
…e Dags Follow-up to apache#66257 addressing review feedback from @jscheffl: - Expand module docstrings on both example Dags with storyline, scope and no-external-dependencies notes. - Add a Dag-level doc_md so the storyline shows up in the UI Dag details page. - Add task-level docstrings explaining what each step does and how it would map to a production pipeline. - Reference both example Dags from the Example Dag Review Checklist as canonical templates for new tutorial-style examples. Signed-off-by: André Ahlert <andre@aex.partners>
jscheffl
left a comment
There was a problem hiding this comment.
Thanks for following up on the comments and directly raising an improvement! COOL! Faster than I myself can put it to my list as penalty being too slow.
Could you point me at which pre-existing example(s) you had in mind so I can replace them in a separate commit on this branch?
The aim of the Example refurbish was to consolidate the existing examples into the Dags that are more crafted around a storyline and "bit more realistic" Dag scenario. And yes, existing Python Operator / Decorator examples are linkes 12-times. Idea would be (can be also a follow-up PR!) to replace the previous examples and in cases where linked/used/referenced in the documentation point to one of the "new" examples along a storyline. As the Dags around the storyline are a bit larger and should contain (hopefully, but ideas welcome) most of the basic functionality finally (also hopefully) less Dags are needed as examples. We have just too much and too much artifical ones.
If you ever attened a Microsoft training you might know the "Northwind Database" and "Contoso Company" which is used consistently across a lot of training courses. Examples more relaistic to real life and usage.
|
CI is green, merging to leave room for another PR :-D |
Follow-up to #66257 addressing the post-merge review from @jscheffl. His comment landed after the merge, so the items below were not part of the original PR. Drafting this for visibility while I wait on guidance for the consolidation item.
What this PR covers from Jens' list:
doc_mdblock that renders in the UI with the storyline, the per-step description, and a link back to the example Dag review checklist.PythonOperator) now has a docstring explaining what it does and how it would map to a production pipeline.contributing-docs/28_example_dag_review_checklist.rst) now references both files as the canonical templates for new tutorial-style example Dags.What is not in this PR (need your input, @jscheffl):
example_python_decorator.py/example_python_operator.pythat overlap conceptually with the measurement correction ones, butproviders/standard/docs/operators/python.rstreferences them ~12 times asexampleincludetargets, so they are not free to delete. Could you point me at which pre-existing example(s) you had in mind so I can replace them in a separate commit on this branch?Was generative AI tooling used to co-author this PR?