Skip to content

Docs: doc_md + task docstrings for measurement correction example Dags#66707

Merged
jscheffl merged 1 commit into
apache:mainfrom
andreahlert:followup-measurement-correction-docs
May 11, 2026
Merged

Docs: doc_md + task docstrings for measurement correction example Dags#66707
jscheffl merged 1 commit into
apache:mainfrom
andreahlert:followup-measurement-correction-docs

Conversation

@andreahlert
Copy link
Copy Markdown
Contributor

Follow-up to #66257 addressing the post-merge review from @jscheffl. His comment landed after the merge, so the items below were not part of the original PR. Drafting this for visibility while I wait on guidance for the consolidation item.

What this PR covers from Jens' list:

  • Dag Markdown Documentation: both example Dags now define a doc_md block that renders in the UI with the storyline, the per-step description, and a link back to the example Dag review checklist.
  • Pydoc at the task level: every task (TaskFlow and PythonOperator) now has a docstring explaining what it does and how it would map to a production pipeline.
  • Take the new example into the docs: the example Dag review checklist (contributing-docs/28_example_dag_review_checklist.rst) now references both files as the canonical templates for new tutorial-style example Dags.

What is not in this PR (need your input, @jscheffl):

  • Consolidating / removing pre-existing artificial examples. The standard provider has example_python_decorator.py / example_python_operator.py that overlap conceptually with the measurement correction ones, but providers/standard/docs/operators/python.rst references them ~12 times as exampleinclude targets, so they are not free to delete. Could you point me at which pre-existing example(s) you had in mind so I can replace them in a separate commit on this branch?

Was generative AI tooling used to co-author this PR?
  • Yes

…e Dags

Follow-up to apache#66257 addressing review feedback from @jscheffl:

- Expand module docstrings on both example Dags with storyline, scope and
  no-external-dependencies notes.
- Add a Dag-level doc_md so the storyline shows up in the UI Dag details
  page.
- Add task-level docstrings explaining what each step does and how it
  would map to a production pipeline.
- Reference both example Dags from the Example Dag Review Checklist as
  canonical templates for new tutorial-style examples.

Signed-off-by: André Ahlert <andre@aex.partners>
Copy link
Copy Markdown
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for following up on the comments and directly raising an improvement! COOL! Faster than I myself can put it to my list as penalty being too slow.

Could you point me at which pre-existing example(s) you had in mind so I can replace them in a separate commit on this branch?

The aim of the Example refurbish was to consolidate the existing examples into the Dags that are more crafted around a storyline and "bit more realistic" Dag scenario. And yes, existing Python Operator / Decorator examples are linkes 12-times. Idea would be (can be also a follow-up PR!) to replace the previous examples and in cases where linked/used/referenced in the documentation point to one of the "new" examples along a storyline. As the Dags around the storyline are a bit larger and should contain (hopefully, but ideas welcome) most of the basic functionality finally (also hopefully) less Dags are needed as examples. We have just too much and too much artifical ones.

If you ever attened a Microsoft training you might know the "Northwind Database" and "Contoso Company" which is used consistently across a lot of training courses. Examples more relaistic to real life and usage.

@jscheffl
Copy link
Copy Markdown
Contributor

CI is green, merging to leave room for another PR :-D

@jscheffl jscheffl merged commit a9764af into apache:main May 11, 2026
167 of 174 checks passed
@andreahlert andreahlert deleted the followup-measurement-correction-docs branch May 12, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants