[BEAM-11205] Upgrading the Google Cloud Libraries BOM to v22 #15411

suztomo · 2021-08-27T18:09:45Z

This is not urgent. It's nice to have latest dependencies.

Upgrading the Google Cloud Libraries BOM to version 22.0.0 and upgrading associated dependencies.

Linkage Checker

https://cwiki.apache.org/confluence/display/BEAM/Dependency+Upgrades

No new linkage errors. https://gist.github.com/suztomo/a0aaa6a86e191009e055575d60f62545

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

suztomo · 2021-08-27T18:10:18Z

Run Java PostCommit

suztomo · 2021-08-27T18:10:27Z

Run PostCommit_Java_Hadoop_Versions

suztomo · 2021-08-27T18:10:33Z

Run PostCommit_Java_Dataflow

suztomo · 2021-08-27T18:10:38Z

Run PostCommit_Java_DataflowV2

suztomo · 2021-08-27T18:10:45Z

Run Spark ValidatesRunner

suztomo · 2021-08-27T18:10:50Z

Run SQL Postcommit

suztomo · 2021-08-27T18:32:36Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

+        google_cloud_bigtable_client_core           : "com.google.cloud.bigtable:bigtable-client-core:1.23.1",
+        google_cloud_bigtable_emulator              : "com.google.cloud:google-cloud-bigtable-emulator:0.138.0",


I just picked the latest versions for bigtable-client-core and google-cloud-bigtable-emulator.

The Libraries BOM 22.0.0 includes the Google Cloud BOM 0.160.0, which includes

com.google.cloud:google-cloud-bigtable:2.0.1 com.google.cloud:google-cloud-bigtable-emulator:0.137.1

So apparently google-cloud-bigtable-emulator should be 0.137.1. But how about bigtable-client-core?

suztomo · 2021-08-27T18:44:53Z

Run Java PostCommit

suztomo · 2021-08-27T18:45:00Z

Run PostCommit_Java_Hadoop_Versions

suztomo · 2021-08-27T18:45:06Z

Run PostCommit_Java_Dataflow

suztomo · 2021-08-27T18:45:12Z

Run PostCommit_Java_DataflowV2

suztomo · 2021-08-27T18:45:18Z

Run Spark ValidatesRunner

suztomo · 2021-08-27T18:45:23Z

Run SQL Postcommit

suztomo · 2021-08-27T18:45:47Z

Run GoPortable PreCommit

suztomo · 2021-08-27T18:45:52Z

Run Python_PVR_Flink PreCommit

suztomo · 2021-08-27T19:08:56Z

Run SQL PreCommit failure

https://ci-beam.apache.org/job/beam_PreCommit_SQL_Commit/4072/console

14:38:12 > Task :sdks:java:testing:tpcds:compileJava
14:38:12 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_SQL_Commit/src/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java:213: error: [argument.type.incompatible] incompatible argument for parameter doc of createRecord.
14:38:12         org.apache.avro.Schema.createRecord(schema.getName() + "_projected", null, null, false);
14:38:12                                                                              ^
14:38:12   found   : null (NullType)
14:38:12   required: @Initialized @NonNull String
14:38:12 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_SQL_Commit/src/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java:213: error: [argument.type.incompatible] incompatible argument for parameter namespace of createRecord.
14:38:12         org.apache.avro.Schema.createRecord(schema.getName() + "_projected", null, null, false);
14:38:12                                                                                    ^
14:38:12   found   : null (NullType)
14:38:12   required: @Initialized @NonNull String
14:38:12 2 errors

This PR is not touching Avro version (1.8.2). Why did it fail?

suztomo · 2021-08-27T19:08:59Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2021-08-27T19:09:57Z

Dataflow V2 Java Post Commit Tests

14:48:06 RuntimeError: ('2 error(s) occurred.', ['**************************************** Licenses were not able to be pulled automatically for some dependencies. Please search source code of the dependencies on the internet and add "license" and "notice" (if available) field to /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_DataflowV2_PR/src/sdks/java/container/license_scripts/dep_urls_java.yaml for each missing license. Dependency List: [jackson-bom-2.12.4,libraries-bom-22.0.0]', '**************************************** License type of some dependencies were not identified. The license type is used to decide whether the source code of the dependency should be pulled or not. Please add "type" field to /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_DataflowV2_PR/src/sdks/java/container/license_scripts/dep_urls_java.yaml for each dependency. Dependency List: [jackson-bom-2.12.4,libraries-bom-22.0.0]'])

suztomo · 2021-08-27T19:18:25Z

Run SQL Postcommit

suztomo · 2021-08-27T19:18:30Z

Run Spark ValidatesRunner

suztomo · 2021-08-27T19:18:38Z

Run Java PostCommit

suztomo · 2021-08-27T19:18:44Z

Run PostCommit_Java_Dataflow

suztomo · 2021-08-27T19:18:50Z

Run PostCommit_Java_DataflowV2

suztomo · 2021-08-27T19:19:07Z

Run PostCommit_Java_Hadoop_Versions

suztomo · 2021-08-27T19:28:20Z

Run CommunityMetrics PreCommit

codecov · 2021-08-27T19:34:39Z

Codecov Report

Merging #15411 (d19ba7a) into master (92b9977) will decrease coverage by 0.04%.
The diff coverage is n/a.

❗ Current head d19ba7a differs from pull request most recent head 9694f70. Consider uploading reports for the commit 9694f70 to get more accurate results

@@            Coverage Diff             @@
##           master   #15411      +/-   ##
==========================================
- Coverage   83.80%   83.75%   -0.05%     
==========================================
  Files         440      442       +2     
  Lines       59886    60050     +164     
==========================================
+ Hits        50186    50294     +108     
- Misses       9700     9756      +56

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/source_test_utils.py	`88.47% <0.00%> (-1.39%)`	⬇️
sdks/python/apache_beam/transforms/combiners.py	`93.59% <0.00%> (-0.12%)`	⬇️
...he_beam/runners/interactive/sql/beam_sql_magics.py	`55.65% <0.00%> (ø)`
...ython/apache_beam/runners/interactive/sql/utils.py	`91.07% <0.00%> (ø)`
...ks/python/apache_beam/runners/interactive/utils.py	`93.70% <0.00%> (+0.10%)`	⬆️
...eam/runners/interactive/interactive_environment.py	`91.07% <0.00%> (+0.74%)`	⬆️
.../python/apache_beam/transforms/periodicsequence.py	`98.36% <0.00%> (+1.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92b9977...9694f70. Read the comment docs.

suztomo · 2021-08-27T20:13:32Z

Run Java PreCommit
https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/18799/console

15:59:00 > Task :sdks:java:io:elasticsearch-tests:elasticsearch-tests-5:test
15:59:00 
15:59:00 org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest > testSplit FAILED
15:59:00     java.lang.AssertionError at ElasticsearchIOTest.java:150

suztomo · 2021-08-27T20:13:37Z

Run Java PreCommit

This also failed.

https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3994/consoleFull

16:30:38 > Task :sdks:java:testing:tpcds:compileJava
16:30:38 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java:213: error: [argument.type.incompatible] incompatible argument for parameter doc of createRecord.
16:30:38         org.apache.avro.Schema.createRecord(schema.getName() + "_projected", null, null, false);
16:30:38                                                                              ^
16:30:38   found   : null (NullType)
16:30:38   required: @Initialized @NonNull String
16:30:38 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java:213: error: [argument.type.incompatible] incompatible argument for parameter namespace of createRecord.
16:30:38         org.apache.avro.Schema.createRecord(schema.getName() + "_projected", null, null, false);
16:30:38                                                                                    ^
16:30:38   found   : null (NullType)
16:30:38   required: @Initialized @NonNull String

suztomo · 2021-08-27T20:14:24Z

Java SDK Post Commit Tests

https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/758/console

15:25:11 > Task :sdks:java:testing:test-utils:compileJava
15:25:11 error: warnings found and -Werror specified
15:25:21 1 error
15:25:21 1 warning

suztomo · 2021-08-27T20:14:33Z

Run Java Postcommit

suztomo · 2021-08-30T01:41:30Z

Run SQL PreCommit

suztomo · 2021-08-31T15:08:16Z

Run SQL PreCommit succeeded https://ci-beam.apache.org/job/beam_PreCommit_SQL_Commit/4086/console on commit d19ba7a . It means httpcore, httpclient, or jackson was the problem.

suztomo · 2021-08-31T15:09:07Z

Well it seems that the problem was fixed in #15419.

suztomo · 2021-08-31T16:10:53Z

Run Java PostCommit

suztomo · 2021-08-31T16:11:00Z

Run PostCommit_Java_Hadoop_Versions

suztomo · 2021-08-31T16:11:06Z

Run PostCommit_Java_Dataflow

suztomo · 2021-08-31T16:11:13Z

Run PostCommit_Java_DataflowV2

suztomo · 2021-08-31T16:11:20Z

Run Spark ValidatesRunner

suztomo · 2021-08-31T16:11:27Z

Run SQL Postcommit

suztomo · 2021-08-31T17:46:19Z

Run Java Precommit

suztomo · 2021-08-31T18:59:28Z

R: @kennknowles , @lukecwik

38 successful checks (including "Java PostCommit", "SQL Post Commit", "Dataflow Java Post Commit", and "Dataflow V2 Java Post Commit")!

This is not urgent. It's just nice to have latest dependencies.

lukecwik · 2021-09-08T17:45:15Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

@@ -530,23 +530,23 @@ class BeamModulePlugin implements Plugin<Project> {
        gax                                         : "com.google.api:gax", // google_cloud_platform_libraries_bom sets version
        gax_grpc                                    : "com.google.api:gax-grpc", // google_cloud_platform_libraries_bom sets version
        gax_httpjson                                : "com.google.api:gax-httpjson", // google_cloud_platform_libraries_bom sets version
-        google_api_client                           : "com.google.api-client:google-api-client:1.31.1", // 1.31.1 is required to run 1.31.0 of google_clients_version below.
+        google_api_client                           : "com.google.api-client:google-api-client:$google_clients_version", // for the libraries using $google_clients_version below.


suztomo force-pushed the libraries-bom-22 branch from f302c8b to 56597e4 Compare August 27, 2021 18:19

suztomo commented Aug 27, 2021

View reviewed changes

[BEAM-11205] Upgrading the Libraries BOM to v22

9694f70

suztomo force-pushed the libraries-bom-22 branch from 56597e4 to 9694f70 Compare August 27, 2021 19:15

suztomo changed the title ~~[BEAM-11205] Upgrading the Libraries BOM to v22~~ [BEAM-11205][not for review yet] Upgrading the Libraries BOM to v22 Aug 31, 2021

suztomo force-pushed the libraries-bom-22 branch from d9f3679 to 9694f70 Compare August 31, 2021 15:11

suztomo changed the title ~~[BEAM-11205][not for review yet] Upgrading the Libraries BOM to v22~~ [BEAM-11205] Upgrading the Libraries BOM to v22 Aug 31, 2021

suztomo changed the title ~~[BEAM-11205] Upgrading the Libraries BOM to v22~~ [BEAM-11205] Upgrading the Google Cloud Libraries BOM to v22 Aug 31, 2021

suztomo marked this pull request as ready for review August 31, 2021 18:55

suztomo requested review from kennknowles and lukecwik August 31, 2021 19:26

lukecwik approved these changes Sep 8, 2021

View reviewed changes

lukecwik merged commit 8367b43 into apache:master Sep 8, 2021

udim mentioned this pull request Sep 30, 2021

Revert "Cherrypick #15418 and #15515" #15629

Merged

		google_cloud_bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.23.1",
		google_cloud_bigtable_emulator : "com.google.cloud:google-cloud-bigtable-emulator:0.138.0",

[BEAM-11205] Upgrading the Google Cloud Libraries BOM to v22 #15411

[BEAM-11205] Upgrading the Google Cloud Libraries BOM to v22 #15411

Conversation

suztomo commented Aug 27, 2021 • edited Loading

Linkage Checker

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo Aug 27, 2021 • edited Loading

Choose a reason for hiding this comment

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021 • edited Loading

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

codecov bot commented Aug 27, 2021 • edited Loading

Codecov Report

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021 • edited Loading

suztomo commented Aug 27, 2021

suztomo commented Aug 27, 2021

suztomo commented Aug 30, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

suztomo commented Aug 31, 2021

lukecwik Sep 8, 2021

Choose a reason for hiding this comment

suztomo commented Aug 27, 2021 •

edited

Loading

`ValidatesRunner` compliance status (on master branch)

suztomo Aug 27, 2021 •

edited

Loading

suztomo commented Aug 27, 2021 •

edited

Loading

codecov bot commented Aug 27, 2021 •

edited

Loading

suztomo commented Aug 27, 2021 •

edited

Loading