[BEAM-8911] New non-vendored Guava version: 25.1-jre by suztomo · Pull Request #10318 · apache/beam

suztomo · 2019-12-06T18:32:11Z

https://issues.apache.org/jira/browse/BEAM-8911

Now unused non-vendored Guava dependencies are removed (BEAM-8747) and Hadoop client version is upgraded to 2.8.5 from 2.7 (BEAM-8822). It's time to upgrade Guava.

New Guava version: 25.1-jre

CharMatcher.DIGIT, used by org.apache.cassandra.io.sstable.format.SSTableFormat, is removed since Guava 26.0. So this PR is setting Guava 25.1:

https://guava.dev/releases/25.1-jre/api/docs/com/google/common/base/CharMatcher.html
https://guava.dev/releases/26.0-jre/api/docs/com/google/common/base/CharMatcher.html

Guava has 2 variations "-jre" for Java8 (stream/lambda support) and "-android" for Java7. Gcsio:1.9.16 needs ImmutableSet.toImmutableSet() which is available in Guava "-jre". detail

So this PR is setting Guava version "25.1-jre".

Which modules use non-vendored Guava?

suztomo@suxtomo24:~/beam6$ find . -name build.gradle | xargs grep library.java.guava
./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
./sdks/java/io/google-cloud-platform/build.gradle:  compile library.java.guava
./sdks/java/io/kinesis/build.gradle:  compile library.java.guava
./sdks/java/io/kinesis/build.gradle:  testCompile library.java.guava_testlib
./sdks/java/extensions/sql/zetasql/build.gradle:  compile library.java.guava

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza	Spark
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

suztomo · 2019-12-06T18:33:27Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

    def google_cloud_spanner_version = "1.6.0"
    def grpc_version = "1.17.1"
-    def guava_version = "20.0"
+    def guava_version = "25.1-jre"


Cassandra-all version 3 does not work with Guava >= 26 (detail)

suztomo · 2019-12-07T03:23:44Z

java.lang.RuntimeException: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/11893 instances, 1/0 CPUs, 250/297535 disk GB, 0/3647 SSD disk GB, 1/68 instance groups, 1/69 managed instance groups, 1/310 instance templates, 1/599 in-use IP addresses.

https://builds.apache.org/job/beam_PostCommit_Java_PR/265/testReport/junit/org.apache.beam.examples.complete/TopWikipediaSessionsIT/testE2ETopWikiPages/

suztomo · 2019-12-08T04:27:55Z

Run Java PostCommit

suztomo · 2019-12-09T02:50:22Z

Run Python PreCommit

suztomo · 2019-12-09T02:52:42Z

R: @kennknowles, @iemejia

Java SDK Post Commit Tests succeeded.

iemejia · 2019-12-09T09:40:52Z

Run Python PreCommit

lgajowy · 2019-12-09T13:19:35Z

Run Java HadoopFormatIO Performance Test

lgajowy · 2019-12-09T13:20:16Z

Run BigQueryIO Streaming Performance Test Java

iemejia · 2019-12-09T13:27:29Z

Run Dataflow ValidatesRunner

iemejia · 2019-12-09T13:27:37Z

Run Spark ValidatesRunner

suztomo · 2019-12-09T14:52:36Z

Run BigQueryIO Streaming Performance Test Java

suztomo · 2019-12-09T15:54:10Z

Run Dataflow ValidatesRunner

suztomo · 2019-12-09T16:49:29Z

10:11:23     SEVERE: 2019-12-09T15:11:22.086Z: An OutOfMemoryException occurred. Consider specifying higher memory instances in PipelineOptions.
10:11:23     java.lang.RuntimeException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
10:11:23     	at org.apache.beam.runners.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntries(BatchingShuffleEntryReader.java:132)

https://builds.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/235/console

It's very red.

suztomo · 2019-12-09T16:49:36Z

Run BigQueryIO Streaming Performance Test Java

suztomo · 2019-12-09T21:12:57Z

Run Dataflow ValidatesRunner

suztomo · 2019-12-09T21:13:04Z

Run BigQueryIO Streaming Performance Test Java

suztomo · 2019-12-10T02:01:17Z

Run Dataflow ValidatesRunner

suztomo · 2019-12-10T04:21:29Z

Run Dataflow ValidatesRunner

lgajowy · 2019-12-10T15:49:14Z

Run Dataflow ValidatesRunner

kennknowles

Once green this seems g2g. The Java postcommits and unit tests of the IOs should be the ones most needed here.

kennknowles · 2019-12-10T20:23:45Z

run sql postcommit

suztomo · 2019-12-10T21:40:42Z

Dataflow Runner Nexmark Tests

suztomo · 2019-12-10T21:40:46Z

Direct Runner Nexmark Tests

suztomo · 2019-12-10T21:40:49Z

Flink Runner Nexmark Tests

suztomo · 2019-12-10T21:40:53Z

Spark Runner Nexmark Tests

suztomo · 2019-12-10T21:40:57Z

Run Java Flink PortableValidatesRunner Batch

suztomo · 2019-12-10T21:41:02Z

Run Java Flink PortableValidatesRunner Streaming

suztomo · 2019-12-10T21:41:16Z

Run Java Spark PortableValidatesRunner Batch

suztomo · 2019-12-10T21:41:20Z

Run Java PortabilityApi PostCommit

suztomo · 2019-12-10T21:41:23Z

Run Java examples on Dataflow with Java 11

suztomo · 2019-12-10T21:41:26Z

Run Java Portability examples on Dataflow with Java 11

suztomo · 2019-12-10T21:41:29Z

Run Apex ValidatesRunner

suztomo · 2019-12-10T21:41:32Z

Run Flink ValidatesRunner

suztomo · 2019-12-10T21:41:36Z

Run Gearpump ValidatesRunner

suztomo · 2019-12-10T21:41:39Z

Run Dataflow PortabilityApi ValidatesRunner

suztomo · 2019-12-10T21:41:43Z

Run Samza ValidatesRunner

suztomo · 2019-12-10T21:41:45Z

Run Spark ValidatesRunner

suztomo · 2019-12-10T21:43:08Z

Dataflow Runner Nexmark Tests

suztomo · 2019-12-10T23:51:00Z

Run Dataflow ValidatesRunner

suztomo · 2019-12-10T23:51:16Z

Dataflow Runner Nexmark Tests

suztomo · 2019-12-11T02:27:44Z

@kennknowles Now they're green; I see some of them didn't trigger builds. Would you merge this?

suztomo · 2019-12-12T18:17:54Z

R: @lgajowy
Found your name in BEAM-5559).

suztomo · 2019-12-12T23:19:51Z

Kenn, thanks!

lgajowy · 2019-12-13T09:49:54Z

Great that all the checks passed. Thank you for doing this @suztomo :)

[BEAM-8911] New Guava version: 25.1-jre

dafb7f2

suztomo commented Dec 6, 2019

View reviewed changes

kennknowles self-requested a review December 10, 2019 20:23

kennknowles approved these changes Dec 10, 2019

View reviewed changes

kennknowles merged commit ab9e8f0 into apache:master Dec 12, 2019

suztomo deleted the guava_26 branch December 12, 2019 23:19

Conversation

suztomo commented Dec 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Guava version: 25.1-jre

Which modules use non-vendored Guava?

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

Uh oh!

suztomo Dec 6, 2019

Choose a reason for hiding this comment

Uh oh!

suztomo commented Dec 7, 2019

Uh oh!

suztomo commented Dec 8, 2019

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

iemejia commented Dec 9, 2019

Uh oh!

lgajowy commented Dec 9, 2019

Uh oh!

lgajowy commented Dec 9, 2019

Uh oh!

iemejia commented Dec 9, 2019

Uh oh!

iemejia commented Dec 9, 2019

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

suztomo commented Dec 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

suztomo commented Dec 9, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

lgajowy commented Dec 10, 2019

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

kennknowles commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

Uh oh!

suztomo commented Dec 10, 2019

suztomo commented Dec 6, 2019 •

edited

Loading

suztomo commented Dec 9, 2019 •

edited

Loading