#10472: Add D200 pydocstyle check - part1 by Samira-g-js · Pull Request #11587 · apache/airflow

Samira-g-js · 2020-10-16T16:14:11Z

Related to #10472 - Pydocstyle check

boring-cyborg · 2020-10-16T16:14:15Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>

- The dag_run argument is only there for test mocks, and only to access a static method. Removing this simplifies the function, reduces confusion. - Give optional arguments a default value, reduce indentation of arg list to PEP / Black standard. - Clean up tests for readability

Although these lists are short, there's no need to re-create them each time, and also no need for them to be a method. I have made them lowercase (`finished`, `running`) instead of uppercase (`FINISHED`, `RUNNING`) to distinguish them from the actual states.

potiuk · 2020-10-16T20:25:58Z

Can you please rebase @Samira-g-js ? there are some conflicts :(

0be7654 commit made an optimization where the plugin are lazy-loaded. However, there are use-cases where you would still want the plugins to be loaded on Airflow start-up.

)

) * raises a warning for provide_context instead of killing the task * Update airflow/operators/python.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * static checks Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

…#11605)

- Sentence completion - code-block rendering - Link to english docs for Azure instructions

`cllient` -> `client` `environement` -> `environment` `naamespace` -> `namespace` `alllow` -> `allow`

* Improves stability of K8S tests by caching binaries and repeats The K8S tests on CI are controlled from the host, not from inside of the CI container image. Therefore it needs virtualenv to run the tests as well as some tools such as helm, kubectl and kind. While those tools can bee downloaded and installed on demand, from time to time the download fails intermittently. This change introduces the following improvements: * the commands to download and setup kind, helm, kubectl are repeated up to 4 times in case they fail * the "bin" directory where those binaries are downloaded is cached between runs. Only the same combination of versions of the tools are sharing the same cache. This way both cases - regular re-runs of the same jobs and upgrade of tools will be much more stable.

Seems like the trap with several steps and || true does not really work the way I wanted and when kill is run but the process is already gone, we had error in the script. Looks like this approach with sub-process kill will do it.

A few remnants of earlier version of the script caused occasional errors. Error introduced in #11541

In case of very simple changes, there might be no merge commits generated by GitHub. In such cases we should take the commit SHA instead as the base of change calculation for selective tests.

…" (#11646) This reverts commit 4fcc71c.

This reverts commit 6fbb235.

…11648) This reverts commit 9237338.

We do not dump airflow logs on success any more, but we dump them and all the container logs in case of failure, so that we can better investigate cases like #11543 - that includes enabling full deadlock information dumping in our mysql database.

Since we support Py 3.6, there is no need of six library

* fix: 🐛 Float to Int columns conversion The `_fix_int_dytpes` method is applying the `astype` transformation to the return of a `np.where` call. I added an extra step to the method in order to apply this to the whole pd.Series. Note that Int64Dtype must be used as an instance, since Pandas will raise an Exception if a class is used. * test: Add dtype test for integers * style: Change line length

…-login is not required. (#11588)

Closed #11388 Co-authored-by: Ryan Hamilton <ryan@ryanahamilton.com>

When we prepare pre-release versions, they are not intended to be converted to final release versions, so there is no need to replace version number for them artificially, For release candidates on the other hand, we should internally use the "final" version because those packages might be simply renamed to the final "production" versions. Fixes #11585

…ow into py-docstyle-check-D200

Samira-g-js · 2020-10-19T11:22:12Z

@potiuk Hey I've rebased with my changes on top, can we just double check that no one else's changes since have been overwritten. I'm a bit of a newbie :(

github-actions · 2020-10-19T12:42:38Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-10-19T12:43:29Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

kaxil · 2020-11-04T00:27:24Z

Can you please rebase your PR on latest Master since we have applied Black and PyUpgrade on Master.

It will help if your squash your commits into single commit first so that there are less conflicts.

Add D200 pydocstyle check

4d8f761

boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:dev-tools area:Scheduler including HA (high availability) scheduler area:secrets area:webserver Webserver related Issues provider:amazon AWS/Amazon - related issues labels Oct 16, 2020

mik-laj and others added 7 commits October 16, 2020 18:28

Add DataflowStartFlexTemplateOperator (#8550)

3c10ca6

Add type annotations for AWS operators and hooks (#11434)

0823d46

Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>

Pass SQLAlchemy engine options to FAB based UI (#11395)

91484b9

Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>

Fix tooltip typo (#11593)

84c7028

Fix broken migration for Sqlite3 (#11573)

44031bf

potiuk approved these changes Oct 16, 2020

View reviewed changes

kaxil and others added 12 commits October 16, 2020 22:56

Allow loading plugins on Airflow start-up (#11596)

be72817

0be7654 commit made an optimization where the plugin are lazy-loaded. However, there are use-cases where you would still want the plugins to be loaded on Airflow start-up.

UX Enhancement: Add button to clear search query from DAG search (#11583

4582c80

)

Set doc_md when using task decorator and function has __doc__ (#11598)

84dc2fb

Cross Reference XCom in tutorial (#11600)

bb615f2

Minor improvements to docs/tutorial.rst (#11604)

b718b51

Remove unnecessary use of comprehension in setup_provider_packages.py (…

55a4d8c

…#11605)

Fix tcp keepalive parameters parsing (#11594)

210a948

Fix rendering of code-block in operator/kubernetes.rst (#11606)

b6ab0b5

Fix Task definition in docs (#11601)

673386d

Minor doc improvements in blob_storage_to_gcs.rst (#11607)

ee9bdff

- Sentence completion - code-block rendering - Link to english docs for Azure instructions

Remove redundant code to serialized k8s.V1Pod (#11602)

16ed974

kaxil and others added 13 commits October 18, 2020 21:12

Fix minor typos in tests (#11638)

885db90

`cllient` -> `client` `environement` -> `environment` `naamespace` -> `namespace` `alllow` -> `allow`

Fixed an error introduced in selective checks (#11640)

6fbb235

A few remnants of earlier version of the script caused occasional errors. Error introduced in #11541

Fixes selective tests in case of missing merge commits (#11641)

4fcc71c

In case of very simple changes, there might be no merge commits generated by GitHub. In such cases we should take the commit SHA instead as the base of change calculation for selective tests.

Revert "Fixes selective tests in case of missing merge commits (#11641)…

c06addf

…" (#11646) This reverts commit 4fcc71c.

Revert "Fixed an error introduced in selective checks (#11640)" (#11647)

8ed2229

This reverts commit 6fbb235.

Revert "Optimizes CI builds heavily with selective checks (#11541)" (#…

d93b6e5

…11648) This reverts commit 9237338.

Remove usage of six (#11645)

fd8b07c

Since we support Py 3.6, there is no need of six library

fix cloudwatch and wasb taskhandler log path config (#11650)

86725f4

Use Python 3 Style super calls (#11644)

89e5acc

Samira-g-js changed the title ~~#10472: Add D200 pydocstyle check~~ #10472: Add D200 pydocstyle check - part1 Oct 19, 2020

alxdembo and others added 8 commits October 19, 2020 10:57

Sourcing the profile file should be sufficient to update the PATH, re…

73b0991

…-login is not required. (#11588)

Add ability to zoom into Graph view from selection in Tree View (#11553)

86ed7dc

Closed #11388 Co-authored-by: Ryan Hamilton <ryan@ryanahamilton.com>

Allow null schedule_interval in OpenAPI spec for DAGs (#11532)

7206fd7

Add optional session argument to xcom_push. (#11485)

22f6db7

Add Plugins View in web UI (#10770)

91898e8

Add D200 pydocstyle check

39765be

Merge branch 'py-docstyle-check-D200' of github.com:Samira-g-js/airfl…

0b7ad05

…ow into py-docstyle-check-D200

mik-laj removed the area:API Airflow's REST/HTTP API label Oct 19, 2020

Samira-g-js marked this pull request as ready for review October 20, 2020 10:56

Samira-g-js closed this Nov 17, 2020

Samira-g-js deleted the py-docstyle-check-D200 branch November 17, 2020 10:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

#10472: Add D200 pydocstyle check - part1#11587

#10472: Add D200 pydocstyle check - part1#11587
Samira-g-js wants to merge 59 commits intoapache:masterfrom
Samira-g-js:py-docstyle-check-D200

Samira-g-js commented Oct 16, 2020

Uh oh!

boring-cyborg bot commented Oct 16, 2020

Uh oh!

potiuk commented Oct 16, 2020

Uh oh!

Samira-g-js commented Oct 19, 2020

Uh oh!

github-actions bot commented Oct 19, 2020

Uh oh!

github-actions bot commented Oct 19, 2020

Uh oh!

kaxil commented Nov 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Comments

Conversation

Samira-g-js commented Oct 16, 2020

Uh oh!

boring-cyborg bot commented Oct 16, 2020

Uh oh!

potiuk commented Oct 16, 2020

Uh oh!

Samira-g-js commented Oct 19, 2020

Uh oh!

github-actions bot commented Oct 19, 2020

Uh oh!

github-actions bot commented Oct 19, 2020

Uh oh!

kaxil commented Nov 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants