Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch #17259

Closed
wants to merge 1 commit into from
Closed

[DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch #17259

wants to merge 1 commit into from

Conversation

youngoli
Copy link
Contributor

@youngoli youngoli commented Apr 4, 2022

You can run many tests automatically using release/src/main/scripts/mass_comment.py.

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run CommunityMetrics PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow Runner Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow Runner V2 Java 11 Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow Runner V2 Java 17 Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow Runner V2 Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow Streaming ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow ValidatesRunner Java 11

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow ValidatesRunner Java 17

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Dataflow ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Direct Runner Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Direct ValidatesRunner Java 11

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Direct ValidatesRunner Java 17

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Direct ValidatesRunner in Java 11

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Direct ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Flink Runner Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Flink ValidatesRunner Java 11

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Flink ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Go Flink ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Go PostCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Go PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Go Samza ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Go Spark ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run GoPortable PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Java 11 Examples on Dataflow Runner V2

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Java 17 Examples on Dataflow Runner V2

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Java Dataflow V2 ValidatesRunner Streaming

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Java Dataflow V2 ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Java Examples on Dataflow Runner V2

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Java Examples_Direct

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python 3.8 PostCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python 3.9 PostCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Dataflow V2 ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Dataflow ValidatesContainer

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Dataflow ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Examples_Dataflow

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Examples_Direct

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Examples_Flink

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Examples_Spark

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Flink ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Samza ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python Spark ValidatesRunner

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run PythonDocker PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run PythonDocs PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run PythonFormatter PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run PythonLint PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Python_PVR_Flink PreCommit

@codecov
Copy link

codecov bot commented Apr 4, 2022 •
edited

Codecov Report

Merging #17259 (92ca0bd) into release-2.38.0 (47a9d8f) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##           release-2.38.0   #17259      +/-   ##
==================================================
- Coverage           73.96%   73.96%   -0.01%     
==================================================
  Files                 672      672              
  Lines               88259    88270      +11     
==================================================
+ Hits                65279    65287       +8     
- Misses              21867    21870       +3     
  Partials             1113     1113
Flag Coverage Δ
python 83.63% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...che_beam/runners/interactive/interactive_runner.py 92.85% <0.00%> (-0.72%) ⬇️
...hon/apache_beam/runners/worker/bundle_processor.py 93.26% <0.00%> (-0.25%) ⬇️
.../apache_beam/options/pipeline_options_validator.py 98.90% <0.00%> (+0.03%) ⬆️
...dks/python/apache_beam/options/pipeline_options.py 94.91% <0.00%> (+0.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47a9d8f...92ca0bd. Read the comment docs.

@youngoli
Copy link
Contributor Author

youngoli commented Apr 4, 2022

Run Flink ValidatesRunner Java 11

@youngoli
Copy link
Contributor Author

youngoli commented Apr 5, 2022

Run Dataflow Runner V2 Nexmark Tests

@youngoli
Copy link
Contributor Author

youngoli commented Apr 5, 2022

Run Dataflow ValidatesRunner Java 17

@youngoli
Copy link
Contributor Author

youngoli commented Apr 5, 2022 •
edited

Logging my triaging on all the failing targets for transparency.

Non-release blocking failures:
CommunityMetrics - Problem in our test infrastructure preventing metrics from being uploaded. Unrelated to Beam itself.
Google Cloud Dataflow Portability Runner Examples Java 11 - False negative. The build is successful but no tests are run and Jenkins reports that as a failure.
[Google Cloud Dataflow Runner V2 Examples](Google Cloud Dataflow Runner V2 Examples) - False negative, same as above.
Java Flink PortableValidatesRunner Streaming Tests - Only error has apparently been failing since being added, which is not a significant regression and therefore not release-blocking. BEAM-14254
Java Samza PortableValidatesRunner Tests - Only failure is the same as the Flink one above, and it's because Samza doesn't support the feature being tested.
Java Spark v2 PortableValidatesRunner Streaming Tests - Not considered release-blocking since the test suite was never consistently green to begin with. See BEAM-13950.
Python 3.7 PostCommit - Almost certainly a test configuration error and probably a flake. Failing to connect to a bigquery table. I'll be trying to deflake this just in case.
PythonDocs PreCommit - Fixed by #17176, I missed that during cherrypicks. Easy fix.

Further investigation needed:
Dataflow Java Post Commit Tests - Many failures. Most seem likely to be non-release blocking, but still looking into it. BEAM-14252
Dataflow V2 Java Post Commit Tests - Fewer failures than above, but some are identical failures and covered by the JIRA above. Then there's two spanner test failures that seem to be due to an incorrect config. Only new failure is not release-blocking (BEAM-14263).
JPMS Java 11 Dataflow Post Commit Tests and JPMS Java 17 Dataflow Post Commit Tests - Failing due to lack of Dataflow worker container for 2.38.0. Will need to rerun once it's available.
Java PreCommit - Been constantly timing out recently and being very flaky. Will need to keep rerunning until it doesn't timeout to see if it's actually failing or not.
Java SDK Post Commit Tests - Same failures as the Dataflow Java Post Commit Tests, so investigating BEAM-14252 will cover this.

@youngoli
Copy link
Contributor Author

youngoli commented Apr 6, 2022

Run Java PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 6, 2022

Run PostCommit_Java_DataflowV2

@youngoli
Copy link
Contributor Author

youngoli commented Apr 6, 2022

Run PostCommit_Java_Dataflow

@youngoli
Copy link
Contributor Author

youngoli commented Apr 6, 2022

Run Java PreCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 6, 2022

Run Java PostCommit

@youngoli
Copy link
Contributor Author

youngoli commented Apr 6, 2022 •
edited

Resolved:
Dataflow Java Post Commit Tests - All bugs on this have been addressed and determined to be non-release blocking. (BEAM-14252)
Dataflow V2 Java Post Commit Tests - Same as above, resolved with BEAM-14252.
Java SDK Post Commit Tests - I forgot to rerun this, but all its failures were covered by BEAM-14252 so it should be resolved now. Will double-check once it's done running.
Java PreCommit - Fixed with cherry-pick #17295.

@youngoli youngoli closed this Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@youngoli