Persist DAG and task doc values in TaskFlow API if explicitly set #29399
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
josh-fell
commented
Feb 7, 2023
| if self.function.__doc__: | ||
| op_doc_attrs = [op.doc, op.doc_json, op.doc_md, op.doc_rst, op.doc_yaml] | ||
| # Set the task's doc_md to the function's docstring if it exists and no other doc* args are set. | ||
| if self.function.__doc__ and not any(op_doc_attrs): |
There was a problem hiding this comment.
Checking against all of the doc attrs for tasks rather than just doc_md. Thinking users most likely wouldn't set more than one of these at a time.
josh-fell
commented
Feb 7, 2023
| @@ -2565,58 +2565,49 @@ def noop_pipeline(): | |||
|
|
|||
| dag = noop_pipeline() | |||
| assert isinstance(dag, DAG) | |||
| assert dag.dag_id, "noop_pipeline" | |||
There was a problem hiding this comment.
Typical example of improper assertion that is being fixed in this PR.
josh-fell
commented
Feb 7, 2023
|
|
||
| task | ||
| @dag_decorator( | ||
| "test-dag", start_date=DEFAULT_DATE, template_searchpath=template_dir, doc_md=template_file |
There was a problem hiding this comment.
This test was broken even after correcting the assertion. Just needed a proper template_searchpath arg.
josh-fell
force-pushed
the
deco-docstrings
branch
from
86962df
to
23ceb45
Compare
If users set `doc_md` arg on `@dag`- or `@task`-decorated TaskFlow functions and those functions have a docstring, the `doc_md` value is not respected. Instead this PR will enforce only using the TaskFlow function docstring as the documentation if `doc_md` (or any `doc*` attrs for tasks) is not set.
josh-fell
force-pushed
the
deco-docstrings
branch
from
23ceb45
to
0eb2c5e
Compare
potiuk
approved these changes
Feb 20, 2023
pierrejeambrun
pushed a commit
that referenced
this pull request
Mar 7, 2023
…9399) If users set `doc_md` arg on `@dag`- or `@task`-decorated TaskFlow functions and those functions have a docstring, the `doc_md` value is not respected. Instead this PR will enforce only using the TaskFlow function docstring as the documentation if `doc_md` (or any `doc*` attrs for tasks) is not set. (cherry picked from commit e02bfc8)
pierrejeambrun
pushed a commit
that referenced
this pull request
Mar 8, 2023
…9399) If users set `doc_md` arg on `@dag`- or `@task`-decorated TaskFlow functions and those functions have a docstring, the `doc_md` value is not respected. Instead this PR will enforce only using the TaskFlow function docstring as the documentation if `doc_md` (or any `doc*` attrs for tasks) is not set. (cherry picked from commit e02bfc8)
64 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If users set
doc_mdarg on@dag- or@task-decorated TaskFlow functions and those functions have a docstring, thedoc_mdvalue is not respected. Instead this PR will enforce only using the TaskFlow function docstring as the documentation ifdoc_md(or anydoc*attrs for tasks) is not set.P.S. There were a number of DAG decorator tests that had improper assertions. Those have been fixed in this PR as well.