[BEAM-11626] Upgrading Guava to 30.1-jre while keeping 25.1-jre for Hadoop/Cassandra modules #13804

suztomo · 2021-01-25T16:29:12Z

This PR upgrades the non-vendored Guava version to the latest 30.1-jre, while keeping the version 25.1-jre for certain modules (Hadoop and Cassandra-related) that require the old version of Guava.

Why do I want the latest Guava?

When Beam publishes a recommended version of Guava for Dataflow users (#13737, WIP), I want the recommended version in line with the one in the GCP Libraries BOM (with "-jre" suffix). This is because Google Cloud client libraries are built and tested with the newer version of Guava. I want Beam's Dataflow and Google Cloud Platform modules to be built, tested, and used with the same version of Guava as much as possible.

If we don't do this PR, we would end up a situation where the GCP Libraries BOM recommends to use Guava 30 and Beam's GCP BOM recommends Guava 25.

What is the problem with Guava 25?

When a library touch classes or methods that only exist in the newer version of Guava, it fails with NoClassDefFoundError or NoSuchMethodError. For example, gcsio uses Uninterruptibles.sleepUninterruptibly(java.time.Duration) in it and Linkage Checker detects the usage:

(com.google.guava:guava:25.1-jre) com.google.common.util.concurrent.Uninterruptibles's method sleepUninterruptibly(java.time.Duration) is not found;
  referenced by 3 class files
    com.google.cloud.hadoop.gcsio.cooplock.CoopLockOperationDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
    com.google.cloud.hadoop.gcsio.cooplock.CoopLockRecordsDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
    com.google.cloud.hadoop.gcsio.testing.InMemoryObjectEntry (com.google.cloud.bigdataoss:gcsio:2.1.6)

The method only exists in Guava 28 or higher. This might not be a problem for Dataflow-only users for now, but this may cause other use cases of the library. Therefore, I want to recommend the newer version of Guava to GCP users.

Problem with newer Guava version in Hadoop/Cassandra

If I naively upgrade the Guava version to 30.1-jre, the tests failed with NoSuchMethodError for Futures.addCallback and
NoSuchFieldError for DIGIT (CharMatcher). Details are in BEAM-11626.

This PR fixes the problem by keeping the Guava version lower for the Hadoop/Cassandra-related modules.

Where is the Guava dependency declared?

The following Gradle modules declare dependency to the guava variable:

suztomo-macbookpro44% find . -name 'build.gradle' |xargs grep 'library.java.guava'
./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
./sdks/java/io/kinesis/build.gradle:  compile library.java.guava
./sdks/java/io/kinesis/build.gradle:  testCompile library.java.guava_testlib
./sdks/java/io/amazon-web-services2/build.gradle:  testCompile library.java.guava_testlib
./sdks/java/io/google-cloud-platform/build.gradle:  compile library.java.guava
./sdks/java/io/contextualtextio/build.gradle:    testCompile library.java.guava_testlib
./sdks/java/extensions/sql/zetasql/build.gradle:  compile library.java.guava
./sdks/java/maven-archetypes/examples/build.gradle:    'guava.version': dependencies.create(project.library.java.guava).getVersion(),
./runners/google-cloud-dataflow-java/build.gradle:  testCompile library.java.guava_testlib

Other than tests, the 3 modules declaring the Guava dependencies are sdks/java/io/kinesis, sdks/java/io/google-cloud-platform, and sdks/java/extensions/sql/zetasql.

sdks/java/io/kinesis module has
- com.amazonaws:amazon-kinesis-client:1.13.0 built with Guava 26.0-jre
- com.amazonaws:amazon-kinesis-producer:0.14.1 built with Guava 24.1.1-jre
- Linkage Checker found no new linkage errors for beam-sdks-java-io-kinesis (link).
Google Cloud libraries are built with the latest Guava
The zetasql client 2020.10.1 is built with Guava 29.
- Linkage Checker detected a potential conflict between org.apache.hadoop:hadoop-yarn-common:2.10.1 (Yarn's WebApp class) and Guava 30. The conflict already exists in Guava 29. Therefore, existing Dataflow Zetasql users will not see a problem with Guava 30.

The sdks/java/maven-archetypes/examples module is tricky one. I want Hadoop/Cassandra users to use Guava 25.1 and others to use Guava 30.

What's the impact to Beam's Cassandra / Hadoop users?

There's no impact to the Beam Cassandra and Hadoop artifacts. The Maven artifact org.apache.beam:beam-sdks-java-io-hadoop-format:2.27.0, org.apache.beam:beam-sdks-java-io-cassandra:2.27.0, or org.apache.beam:beam-sdks-java-io-hadoop-file-system:2.27.0
does not declare Guava dependency.

Instruction for Hadoop / Cassandra Beam users

If Beam Cassandra / Hadoop users use Beam with beam-sdks-java-io-kinesis, beam-sdks-java-io-google-cloud-platform, or beam-sdks-java-extensions-sql-zetasql, then the users need to pin Guava version to 25.1-jre. They can use <dependencyManagement> for Maven and force for Gradle.

Linkage Check

https://gist.github.com/suztomo/beb9033d39da1545d117750c0602ae69

No new linkage errors for the default list. There is one error in Yarn's WebApp class (see above), which is a provided dependency of hadoop-client.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-01-25T16:47:11Z

Codecov Report

Merging #13804 (39c7369) into master (3930d12) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #13804      +/-   ##
==========================================
- Coverage   82.97%   82.95%   -0.03%     
==========================================
  Files         469      469              
  Lines       58343    58343              
==========================================
- Hits        48411    48399      -12     
- Misses       9932     9944      +12

Impacted Files	Coverage Δ
...hon/apache_beam/runners/direct/test_stream_impl.py	`91.91% <0.00%> (-2.21%)`	⬇️
...ks/python/apache_beam/runners/worker/data_plane.py	`89.52% <0.00%> (-1.80%)`	⬇️
sdks/python/apache_beam/io/iobase.py	`84.55% <0.00%> (-0.27%)`	⬇️
sdks/python/apache_beam/transforms/util.py	`95.66% <0.00%> (-0.18%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.86% <0.00%> (-0.13%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3930d12...39c7369. Read the comment docs.

suztomo · 2021-01-25T17:45:45Z

Run SQL Postcommit

suztomo · 2021-01-25T17:45:51Z

Run Spark ValidatesRunner

suztomo · 2021-01-25T17:45:57Z

Run Dataflow ValidatesRunner

suztomo · 2021-01-25T17:46:04Z

Run Java HadoopFormatIO Performance Test

suztomo · 2021-01-25T17:46:16Z

Run Java PostCommit

suztomo · 2021-01-25T17:46:22Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2021-01-25T22:31:47Z

34 successful and 1 skipped checks at 15d7a3d.

suztomo · 2021-01-26T19:42:19Z

Run Java PreCommit

suztomo · 2021-01-26T19:42:27Z

Run SQL Postcommit

suztomo · 2021-01-26T19:42:35Z

Run Spark ValidatesRunner

suztomo · 2021-01-26T19:42:43Z

Run Dataflow ValidatesRunner

suztomo · 2021-01-26T19:42:51Z

Run Java HadoopFormatIO Performance Test

suztomo · 2021-01-26T19:43:08Z

Run Java PostCommit

suztomo · 2021-01-26T21:03:54Z

Run SQL Postcommit

suztomo · 2021-01-26T21:42:23Z

Run SQL Postcommit

ibzib · 2021-01-27T01:03:24Z

Run SQL Postcommit

ibzib

I guess this is still a draft, but LGTM while I'm here :)

suztomo · 2021-01-27T01:29:47Z

@ibzib Thanks. I see all checks are green. Moved this PR out of draft.

suztomo · 2021-01-27T01:40:17Z

R: @ibzib @kennknowles @aaltay

For #13740 (comment) by @aaltay

Ack. I think this would be an undocumented hurdle for the impacted users. I am not sure what is the best course of action. Hopefully @kennknowles would have a recommendation.

Let me know if I need to add more information somewhere.

…ava_force

aaltay · 2021-02-03T20:38:53Z

R: @ibzib @kennknowles @aaltay

For #13740 (comment) by @aaltay

Ack. I think this would be an undocumented hurdle for the impacted users. I am not sure what is the best course of action. Hopefully @kennknowles would have a recommendation.

Let me know if I need to add more information somewhere.

We can add a note to the release note about this.

I think a more sustainable solution would be to add a page java sdk, its dependencies, and addressing common issues related to dependencies. (@kennknowles and @tysonjh - would such a page be helpful to users?)

tysonjh · 2021-02-12T20:51:54Z

R: @ibzib @kennknowles @aaltay
For #13740 (comment) by @aaltay

Ack. I think this would be an undocumented hurdle for the impacted users. I am not sure what is the best course of action. Hopefully @kennknowles would have a recommendation.

Let me know if I need to add more information somewhere.

We can add a note to the release note about this.

I think a more sustainable solution would be to add a page java sdk, its dependencies, and addressing common issues related to dependencies. (@kennknowles and @tysonjh - would such a page be helpful to users?)

I think so. There isn't much developer content that would cover this area in the existing website (e.g. https://beam.apache.org/documentation/sdks/java/). It could be a place to host a FAQ or best practices that could include this information, recommending using BOMs where possible, etc.

A dependency issue with Guava is sadly a pretty well discussed problem throughout the web so developers are pretty familiar with the steps for resolution.

…ava_force

suztomo · 2021-02-12T22:12:49Z

Run SQL Postcommit

suztomo · 2021-02-12T22:13:02Z

Run Spark ValidatesRunner

suztomo · 2021-02-12T22:13:12Z

Run Dataflow ValidatesRunner

suztomo · 2021-02-12T22:13:21Z

Run Java HadoopFormatIO Performance Test

aaltay · 2021-02-12T22:30:51Z

R: @ibzib @kennknowles @aaltay
For #13740 (comment) by @aaltay

Ack. I think this would be an undocumented hurdle for the impacted users. I am not sure what is the best course of action. Hopefully @kennknowles would have a recommendation.

Let me know if I need to add more information somewhere.

We can add a note to the release note about this.
I think a more sustainable solution would be to add a page java sdk, its dependencies, and addressing common issues related to dependencies. (@kennknowles and @tysonjh - would such a page be helpful to users?)

I think so. There isn't much developer content that would cover this area in the existing website (e.g. beam.apache.org/documentation/sdks/java). It could be a place to host a FAQ or best practices that could include this information, recommending using BOMs where possible, etc.

A dependency issue with Guava is sadly a pretty well discussed problem throughout the web so developers are pretty familiar with the steps for resolution.

Got it. It is your call as you are all more familiar with Java specific concerns.

suztomo · 2021-02-16T15:37:05Z

Run SQL Postcommit

suztomo · 2021-02-16T15:37:12Z

Run Java PostCommit

suztomo · 2021-02-16T15:37:17Z

Run Spark ValidatesRunner

suztomo · 2021-02-16T15:37:23Z

Run Dataflow ValidatesRunner

suztomo · 2021-02-16T15:37:32Z

Run Java HadoopFormatIO Performance Test

suztomo · 2021-02-16T17:02:32Z

Run Python PreCommit

suztomo · 2021-02-17T15:55:45Z

@aaltay It's ready for merge; 34 successful checks, including "Java PostCommit".

so developers are pretty familiar with the steps for resolution.

Yes, I also observe the same. For Beam users, this is just a normal version upgrade of transitive Guava dependency. I've added an item into the "Breaking Changes" section of CHANGES.md to make it easy to identify when this change happened.

kennknowles

This is probably the best PR description I have ever read. Thank you!

kennknowles · 2021-02-17T16:54:35Z

I agree with @suztomo that generally users know about the issues with Guava. While technically a breaking change (as many dep upgrades are) I think this will improve things greatly for users and will fix potentially surprising bugs.

iemejia · 2021-02-19T19:55:04Z

Sorry to hijack this PR but I wanted to ask @suztomo or someone else to take care of upgrading our vendored gRPC. It is probably a good idea because of the security issue mentioned in https://issues.apache.org/jira/browse/BEAM-11227 Can someone please take a look

suztomo · 2021-02-19T20:34:29Z

@iemejia Let me create a PR to see what breaks. (I don't know how it's released though)

kennknowles · 2021-02-22T22:37:05Z

Thanks @suztomo! If you can get it upgraded and tested I can kick off a release. I don't recall exactly, but I think I can figure it out and document it.

suztomo · 2021-02-22T23:14:32Z

@kennknowles I think I found the document. https://github.com/apache/beam/tree/master/vendor

Now questions is how we confirm the CVE is addressed https://issues.apache.org/jira/browse/BEAM-11227?focusedCommentId=17287496&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17287496 (let's continue comments there)

configuration.testRuntime for Hadoop modules

a0cefed

suztomo marked this pull request as draft January 25, 2021 16:55

'all' configuration

15d7a3d

suztomo changed the title ~~[BEAM-11626] Configuration.testRuntime to have Guava 25.1 for Hadoop modules~~ [BEAM-11626] Upgrading Guava to 30.1-jre while keeping 25.1-jre for Hadoop/Cassandra modules Jan 25, 2021

Note in CHANGES.md

c85793e

suztomo mentioned this pull request Jan 25, 2021

[BEAM-11626] Guava version 25.1-jre for Hadoop/Cassandra and Guava version 30.1 (latest) for the rest #13740

Closed

4 tasks

Merge remote-tracking branch 'origin/master' into guava_force

bda2a33

suztomo mentioned this pull request Jan 26, 2021

[BEAM-10925] Load SQL UDFs from jar. #13629

Merged

4 tasks

ibzib approved these changes Jan 27, 2021

View reviewed changes

suztomo marked this pull request as ready for review January 27, 2021 01:29

Merge remote-tracking branch 'origin/master' into guava_force

e1635da

Merge branch 'guava_force' of https://github.com/suztomo/beam into gu…

3a6aa0a

…ava_force

suztomo added 2 commits February 12, 2021 17:09

Merge remote-tracking branch 'origin/master' into guava_force

46cd3f1

Merge branch 'guava_force' of https://github.com/suztomo/beam into gu…

6ca9251

…ava_force

Merge remote-tracking branch 'origin/master' into guava_force

39c7369

kennknowles approved these changes Feb 17, 2021

View reviewed changes

kennknowles merged commit 8f87957 into apache:master Feb 17, 2021

TheNeuralBit mentioned this pull request Feb 17, 2021

[BEAM-11805] Replace user-agent for spanner #13990

Merged

4 tasks

[BEAM-11626] Upgrading Guava to 30.1-jre while keeping 25.1-jre for Hadoop/Cassandra modules #13804

[BEAM-11626] Upgrading Guava to 30.1-jre while keeping 25.1-jre for Hadoop/Cassandra modules #13804

Conversation

suztomo commented Jan 25, 2021 • edited

Why do I want the latest Guava?

What is the problem with Guava 25?

Problem with newer Guava version in Hadoop/Cassandra

Where is the Guava dependency declared?

What's the impact to Beam's Cassandra / Hadoop users?

Instruction for Hadoop / Cassandra Beam users

Linkage Check

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Jan 25, 2021 • edited

Codecov Report

suztomo commented Jan 25, 2021

suztomo commented Jan 25, 2021

suztomo commented Jan 25, 2021

suztomo commented Jan 25, 2021

suztomo commented Jan 25, 2021

suztomo commented Jan 25, 2021

suztomo commented Jan 25, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

suztomo commented Jan 26, 2021

ibzib commented Jan 27, 2021

ibzib left a comment

Choose a reason for hiding this comment

suztomo commented Jan 27, 2021

suztomo commented Jan 27, 2021

aaltay commented Feb 3, 2021

tysonjh commented Feb 12, 2021

suztomo commented Feb 12, 2021

suztomo commented Feb 12, 2021

suztomo commented Feb 12, 2021

suztomo commented Feb 12, 2021

aaltay commented Feb 12, 2021

suztomo commented Feb 16, 2021

suztomo commented Feb 16, 2021

suztomo commented Feb 16, 2021

suztomo commented Feb 16, 2021

suztomo commented Feb 16, 2021

suztomo commented Feb 16, 2021

suztomo commented Feb 17, 2021

kennknowles left a comment

Choose a reason for hiding this comment

kennknowles commented Feb 17, 2021

iemejia commented Feb 19, 2021

suztomo commented Feb 19, 2021

kennknowles commented Feb 22, 2021

suztomo commented Feb 22, 2021

suztomo commented Jan 25, 2021 •

edited

codecov bot commented Jan 25, 2021 •

edited