[BEAM-3713] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest #14795

benWize · 2021-05-12T17:21:52Z

Move integration tests from nose to pytest on the following jobs:

job_PerformanceTests_Python
job_PostCommit_CrossLanguageValidatesRunner_Dataflow

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

benWize · 2021-05-12T17:22:03Z

Run XVR_Dataflow PostCommit

codecov · 2021-05-12T17:47:38Z

Codecov Report

Merging #14795 (3fe2dca) into master (00eb420) will increase coverage by 0.21%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14795      +/-   ##
==========================================
+ Coverage   83.78%   83.99%   +0.21%     
==========================================
  Files         435      439       +4     
  Lines       58423    60014    +1591     
==========================================
+ Hits        48948    50408    +1460     
- Misses       9475     9606     +131

Impacted Files	Coverage Δ
.../examples/snippets/transforms/elementwise/pardo.py
...he_beam/io/flink/flink_streaming_impulse_source.py
...hon/apache_beam/runners/direct/direct_userstate.py
..._beam/io/gcp/internal/clients/bigquery/__init__.py
.../python/apache_beam/runners/worker/statesampler.py
...n/apache_beam/runners/interactive/cache_manager.py
...on/apache_beam/runners/direct/helper_transforms.py
...n/apache_beam/examples/complete/game/user_score.py
...38/build/srcs/sdks/python/apache_beam/io/fileio.py
...thon/apache_beam/io/azure/blobstoragefilesystem.py
... and 864 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00eb420...3fe2dca. Read the comment docs.

benWize · 2021-05-12T18:08:32Z

Run XVR_Dataflow PostCommit

aaltay · 2021-05-27T22:53:58Z

What is the next step on this PR?

benWize · 2021-05-28T15:55:12Z

Hi @aaltay!
Some of these tests failed on pytest, I need to investigate further why it is happening. I'll retake this.

benWize · 2021-05-29T00:29:05Z

Run XVR_Dataflow PostCommit

benWize · 2021-05-29T01:56:06Z

Run XVR_Dataflow PostCommit

benWize · 2021-05-31T17:15:38Z

R: @udim, Hi Udi, would you help me to review this?
I left the nosetests prefix to be able to see the JUnit file before merging.
There is a successful run of PostCommit_CrossLanguageValidatesRunner_Dataflow
I think in order to test PerformanceTests_Python it'll be necessary to run a seed job because the test reference is in the Jenkins file

beam/.test-infra/jenkins/job_PerformanceTests_Python.groovy

Line 40 in 1caeef3

    
           test              : "apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it",

udim · 2021-06-02T01:18:57Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

+          "test_opts": config.pytestOptions,
+          "suite": "xlangValidateRunner",
+          "pytest": true, // TODO(BEAM-3713): Remove this once nose is removed.
+          // TODO(BEAM-10507): Remove no_xlang_transforms after fixing


I think you can remove this TODO. We will always want to exclude tests marked with no_xlang_transforms.

udim · 2021-06-02T01:25:48Z

sdks/python/apache_beam/io/external/xlang_parquetio_test.py

@@ -36,7 +36,7 @@
 PARQUET_WRITE_URN = "beam:transforms:xlang:test:parquet_write"


-@attr('UsesCrossLanguageTransforms')
+@pytest.mark.xlang_transforms


Alternatively, you could remove the no_xlang_transforms marker and just comment out this line with the same TODO.

I did this, I removed the no_xlang_transforms marker and commented out those tests

A successful run with changes in marker no_xlang_transforms https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow_PR/23/testReport/

udim · 2021-06-02T01:28:33Z

Run BigQueryIO Read Performance Test Python

udim · 2021-06-02T01:37:04Z

The run of PostCommit_CrossLanguageValidatesRunner_Dataflow looks okay. I ran the seed job on my pr and started a random performance test: https://ci-beam.apache.org/job/beam_PerformanceTests_BiqQueryIO_Read_Python_PR/3/

udim · 2021-06-02T01:47:14Z

Run Python37 WordCountIT Performance Test

udim · 2021-06-02T02:07:25Z

https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/2959/console failed. I think the pytest timeout of 600s might be the issue

…rmanceTest

benWize · 2021-06-02T18:06:47Z

https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/2959/console failed. I think the pytest timeout of 600s might be the issue

I think the same, I increased the timeout

benWize · 2021-06-09T14:07:00Z

Run XVR_Dataflow PostCommit

benWize · 2021-06-09T14:32:58Z

Hi @udim, would you help me to retest the wordcountIT performance test?

tysonjh · 2021-06-11T18:18:36Z

Run Python37 WordCountIT Performance Test

udim · 2021-06-15T19:04:29Z

Run Python37 WordCountIT Performance Test

udim · 2021-06-15T21:07:05Z

run was successful: https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/3002/

udim

Looks good, just needs to resolve the temporary TODO(BEAM-3713)s

benWize · 2021-06-15T22:32:19Z

Thanks, Udi, I removed the TODOs.

udim · 2021-06-16T01:13:41Z

Run Java PreCommit

udim · 2021-06-16T01:13:51Z

run python precommit

udim · 2021-06-17T19:01:02Z

Ignoring the precommit failures that seem unrelated

chamikaramj · 2021-06-21T22:28:44Z

Seems like this broke portable xvr tests.

For example,
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/1361/console

ModuleNotFoundError: No module named 'pytest'

(Flink and Spark test suites are broken as well)

benWize · 2021-06-22T04:13:39Z

Seems like this broke portable xvr tests.

For example,
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/1361/console

ModuleNotFoundError: No module named 'pytest'

(Flink and Spark test suites are broken as well)

Fix in #15058

tvalentyn · 2021-06-29T19:58:44Z

This change might have started to cause test flakiness: https://issues.apache.org/jira/browse/BEAM-12515

udim · 2021-06-29T21:33:09Z

I don't see how this PR would affect how PipelineOptionsTest.test_display_data runs. It was running with pytest before this PR.

benWize · 2021-06-30T01:16:54Z

I don't think this PR is related to that flakiness, I was able to reproduce it locally in code before this PR was merge using this plugin https://github.com/dropbox/pytest-flakefinder as:
pytest -o junit_suite_name=py38-cloud --junitxml=pytest_py38-cloud.xml --log-cli-level=DEBUG "apache_beam/options/pipeline_options_test.py::PipelineOptionsTest" --flake-finder --flake-runs=10 -n 9.
The problem could be related to xdist, maybe adding the marker @pytest.mark.no_xdist to test_display_data should fix the problem.

benWize · 2021-06-30T03:42:10Z

Possible fix for test_display_data flakiness in #15104

benWize added 5 commits May 4, 2021 15:16

[BEAM-3713] Add pytest option to run_integration_tests

33c6159

[BEAM-3713] Move runPerformanceTest from nose to pytest

bed3e90

[BEAM-3713] Move validatesCrossLanguageRunner from nose to pytest

42ac00b

[BEAM-3713] Merge master

1c3cf98

[BEAM-3713] Lint formatting

9881d82

[BEAM-3713] Fix bad substitution on collect markers

3e0f064

benWize added 2 commits May 17, 2021 12:38

Merge branch 'master' into BEAM-3713-performance

000c62a

Merge master

45b788b

benWize added 2 commits May 28, 2021 12:20

Merge branch 'master' into BEAM-3713-performance

8ffc3cd

[BEAM-37137] Add nosetest prefix for testing

1caeef3

benWize changed the title ~~[BEAM-3713][WIP] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest~~ [BEAM-3713] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest May 31, 2021

benWize marked this pull request as ready for review May 31, 2021 16:49

udim reviewed Jun 2, 2021

View reviewed changes

[BEAM-3713] Remove no_xlang_transforms marker, increase timeout perfo…

91df7c3

…rmanceTest

benWize requested a review from udim June 2, 2021 18:12

udim approved these changes Jun 15, 2021

View reviewed changes

[BEAM-3713] Remove temporary prefix used for testing

3fe2dca

udim merged commit 9418994 into apache:master Jun 17, 2021

benWize mentioned this pull request Jun 22, 2021

[BEAM-37137] Fix xlang dependency error #15058

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-3713] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest #14795

[BEAM-3713] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest #14795

benWize commented May 12, 2021

benWize commented May 12, 2021

codecov bot commented May 12, 2021 •

edited

benWize commented May 12, 2021

aaltay commented May 27, 2021

benWize commented May 28, 2021

benWize commented May 29, 2021

benWize commented May 29, 2021

benWize commented May 31, 2021

udim Jun 2, 2021

udim Jun 2, 2021

benWize Jun 2, 2021

benWize Jun 9, 2021

udim commented Jun 2, 2021

udim commented Jun 2, 2021

udim commented Jun 2, 2021

udim commented Jun 2, 2021 •

edited

benWize commented Jun 2, 2021

benWize commented Jun 9, 2021

benWize commented Jun 9, 2021

tysonjh commented Jun 11, 2021

udim commented Jun 15, 2021

udim commented Jun 15, 2021

udim left a comment

benWize commented Jun 15, 2021

udim commented Jun 16, 2021

udim commented Jun 16, 2021

udim commented Jun 17, 2021

chamikaramj commented Jun 21, 2021

benWize commented Jun 22, 2021

tvalentyn commented Jun 29, 2021

udim commented Jun 29, 2021

benWize commented Jun 30, 2021

benWize commented Jun 30, 2021 •

edited

[BEAM-3713] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest #14795

[BEAM-3713] Move PerformanceTest and CrossLanguageValidateRunner from nosetest to pytest #14795

Conversation

benWize commented May 12, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

benWize commented May 12, 2021

codecov bot commented May 12, 2021 • edited

Codecov Report

benWize commented May 12, 2021

aaltay commented May 27, 2021

benWize commented May 28, 2021

benWize commented May 29, 2021

benWize commented May 29, 2021

benWize commented May 31, 2021

udim Jun 2, 2021

Choose a reason for hiding this comment

udim Jun 2, 2021

Choose a reason for hiding this comment

benWize Jun 2, 2021

Choose a reason for hiding this comment

benWize Jun 9, 2021

Choose a reason for hiding this comment

udim commented Jun 2, 2021

udim commented Jun 2, 2021

udim commented Jun 2, 2021

udim commented Jun 2, 2021 • edited

benWize commented Jun 2, 2021

benWize commented Jun 9, 2021

benWize commented Jun 9, 2021

tysonjh commented Jun 11, 2021

udim commented Jun 15, 2021

udim commented Jun 15, 2021

udim left a comment

Choose a reason for hiding this comment

benWize commented Jun 15, 2021

udim commented Jun 16, 2021

udim commented Jun 16, 2021

udim commented Jun 17, 2021

chamikaramj commented Jun 21, 2021

benWize commented Jun 22, 2021

tvalentyn commented Jun 29, 2021

udim commented Jun 29, 2021

benWize commented Jun 30, 2021

benWize commented Jun 30, 2021 • edited

`ValidatesRunner` compliance status (on master branch)

codecov bot commented May 12, 2021 •

edited

udim commented Jun 2, 2021 •

edited

benWize commented Jun 30, 2021 •

edited