Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-7060] Introduce Python3-only test modules #9223

Merged
merged 3 commits into from
Aug 2, 2019
Merged

[BEAM-7060] Introduce Python3-only test modules #9223

merged 3 commits into from
Aug 2, 2019

Conversation

udim
Copy link
Member

@udim udim commented Aug 1, 2019 •
edited
Loading

This is required for testing type annotations support (PEP 484).

Note that *py3.py files are only intended for tests where the syntax is incompatible with Py2. Not intended to replace checks for where certain versions of Python behave differently.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status		 --- Build Status
Build Status		 Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@udim
[BEAM-7060] Introduce Python3-only test modules
2fd9039 
This is required for testing type annotations support (PEP 484).

Also introduces pylinting of modified files, which reduces gradlew lint
times by ~37% (or ~50s).
before: https://scans.gradle.com/s/544uyirsdn6n4
after: https://scans.gradle.com/s/a3sys35gig3kc
@udim
Copy link
Member Author

udim commented Aug 1, 2019

R: @tvalentyn @chamikaramj

@udim
Remove MODIFIED= code
254aa44 
It does not support already commited and possibly pushed but not merged
commits, and even if I added commands to list those for my workflow
there's no guarantee that it'd work for everone else's.
chamikaramj
chamikaramj reviewed Aug 2, 2019
View reviewed changes
Copy link
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

sdks/python/scripts/run_pylint.sh
@@ -60,23 +60,34 @@ EXCLUDED_GENERATED_FILES=(
apache_beam/portability/api/*pb2*.py
)

PYTHON_MAJOR=$(python -c 'import sys; print(sys.version_info[0])')
Copy link
Contributor

@chamikaramj chamikaramj Aug 2, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a JIRA to do lint checks for *py3.py files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done: https://issues.apache.org/jira/browse/BEAM-7884

tvalentyn
tvalentyn reviewed Aug 2, 2019
View reviewed changes
sdks/python/scripts/run_pylint.sh
FILES_TO_IGNORE=""
for file in "${EXCLUDED_GENERATED_FILES[@]}"; do
for file in "${EXCLUDED_GENERATED_FILES[@]}" ${EXCLUDED_PY3_FILES}; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this loop work as intended when $EXCLUDED_PY3_FILES has more than one file?

Alternative solution: https://github.com/apache/beam/pull/8505/files#diff-d91ac373b8d2235683a85576912b5db5R66
which requires: https://github.com/apache/beam/pull/8505/files#diff-d91ac373b8d2235683a85576912b5db5R87

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see: https://scans.gradle.com/s/544uyirsdn6n4/console-log#L206-L209

@tvalentyn
Copy link
Contributor

Run Python 2 PostCommit

@udim
Copy link
Member Author

udim commented Aug 2, 2019

@tvalentyn I believe pre-commits cover all the changes in this PR.

@tvalentyn
Copy link
Contributor

A similar change was reverted because it broke postcommits: #8505 (comment)

@udim
Copy link
Member Author

udim commented Aug 2, 2019

run python precommit

@udim
Copy link
Member Author

udim commented Aug 2, 2019

run python 2 postcommit

@udim
Copy link
Member Author

udim commented Aug 2, 2019

Run Python Dataflow ValidatesContainer

@udim
Copy link
Member Author

udim commented Aug 2, 2019

Error in postcommit is a transient pip issue

@chamikaramj
Copy link
Contributor

LGTM. Thanks.

@udim udim merged commit 149153b into apache:master Aug 2, 2019
@amaliujia
Copy link
Contributor

amaliujia commented Aug 5, 2019 •
edited
Loading

Hello!

I am thinking if this PR causes ./gradlew spotlessApply fail in master branch.

Run ./gradlew spotlessApply gives the following error message on master branch:

* Where:
Build file '/Users/ruwang/Documents/beam/sdks/python/apache_beam/testing/load_tests/build.gradle' line: 44

* What went wrong:
A problem occurred evaluating project ':sdks:python:apache_beam:testing:load_tests'.
> Could not get unknown property 'files' for task ':sdks:python:apache_beam:testing:load_tests:run' of type org.gradle.api.tasks.Exec.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

I am working on a macbook.

@tvalentyn
Copy link
Contributor

Thanks, @amaliujia, I opened https://issues.apache.org/jira/browse/BEAM-7892 to track this.

@tvalentyn tvalentyn mentioned this pull request Aug 6, 2019
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chamikaramj chamikaramj chamikaramj left review comments

@tvalentyn tvalentyn tvalentyn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@udim @tvalentyn @chamikaramj @amaliujia