[BEAM-8691] bigtable-client-core dependency upgrade #10714

suztomo · 2020-01-29T19:02:45Z

https://issues.apache.org/jira/projects/BEAM/issues/BEAM-8691

Upgrading bigtable-client-core to the latest 1.13.0
The exposed API surface reduced. The associated test is updated.
Because bigtable-client-core:1.13.0 introduced conflicts in Maven version range, excluding grpc-core dependencies from bigtable-client-core and google-cloud-core-grpc, so that the module uses grpc version 1.25.0 (detail)
Upgrading gax to avoid linkage errors (detail)
Declaring grpc-grpclb 1.25.0 to avoid linkage errors (detail)

Linkage Check result

Checked beam-sdks-java-core, beam-sdks-java-io-google-cloud-platform, beam-runners-google-cloud-dataflow-java and beam-sdks-java-io-hadoop-format.

https://gist.github.com/suztomo/1bedcd7c63bc88310fff3854c19710d6

For beam-sdks-java-io-google-cloud-platform and beam-runners-google-cloud-dataflow-java

This PR removes linkage errors on gax-httpjson-0.52.0.jar
This PR upgrades the version of appengine-api-1.0-sdk in the errors. These are irrelevant to Beam.

Using this shell script https://gist.github.com/suztomo/ec4834151a81f8e1d26aa0afb278818b

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza	Spark
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

suztomo · 2020-01-29T19:08:50Z

...ava/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java

@@ -61,11 +61,9 @@ public void testGcpApiSurface() throws Exception {
            classesInPackage("com.google.api.client.util"),
            classesInPackage("com.google.api.services.bigquery.model"),
            classesInPackage("com.google.auth"),
-            classesInPackage("com.google.bigtable.admin.v2"),


The exposed API surface reduced.

lukecwik · 2020-01-29T21:47:38Z

ping for tests

lukecwik · 2020-01-29T21:47:51Z

Run Java PostCommit

lukecwik · 2020-01-29T21:47:57Z

Run Java HadoopFormatIO Performance Test

lukecwik · 2020-01-29T21:48:02Z

Run BigQueryIO Streaming Performance Test Java

lukecwik · 2020-01-29T21:48:09Z

Run Dataflow ValidatesRunner

lukecwik · 2020-01-29T21:48:17Z

Run Spark ValidatesRunner

lukecwik · 2020-01-29T21:48:24Z

Run SQL Postcommit

suztomo · 2020-01-29T21:56:09Z

@lukecwik Thank you.

suztomo · 2020-01-30T03:47:49Z

Java precommit failed:

18:48:45 > Task :sdks:java:io:rabbitmq:test FAILED
18:48:45 
18:48:45 FAILURE: Build failed with an exception.
18:48:45 
18:48:45 * What went wrong:
18:48:45 Execution failed for task ':sdks:java:io:rabbitmq:test'.
18:48:45 > Process 'Gradle Test Executor 124' finished with non-zero exit value 143
18:48:45   This problem might be caused by incorrect test process configuration.
18:48:45   Please refer to the test execution section in the User Manual at https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution

Java_Examples_Dataflow failed:

java.lang.RuntimeException: Dataflow job 2020-01-29_13_55_09-15611797686391011440 terminated in state UNRECOGNIZED but did not return a failure reason.

They both seem transient errors.

boyuanzz · 2020-01-30T04:06:09Z

Run Java PreCommit

boyuanzz · 2020-01-30T04:06:19Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2020-01-30T04:17:56Z

@boyuanzz Thank you!

suztomo · 2020-01-30T13:39:12Z

Java precommit failed:

WARNING: Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead.
java.lang.UnsatisfiedLinkError: could not load a native library: netty-transport-native-epoll

Caused by: java.lang.RuntimeException: failed to get field ID: DefaultFileRegion.transfered

https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1686/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/

Java_Examples_Dataflow failed again with the same error.

java.lang.RuntimeException: Dataflow job 2020-01-29_20_12_07-5864892158255540617 terminated in state UNRECOGNIZED but did not return a failure reason.

Sent an email on this to beam dev mailing list.

udim · 2020-01-31T17:27:46Z

Run Java PreCommit

udim · 2020-01-31T17:28:02Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2020-01-31T18:26:25Z

Dataflow example still fails. This time it's "No space left on device":

12:39:08 > Task :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker
12:39:08 java.io.IOException: No space left on device
12:39:08 com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device

https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/268/consoleFull

Is this specific to this PR? I'll confirm the word-count example in my local environment with my branch.

@udim Thank you.

udim · 2020-01-31T19:15:15Z

Dataflow example still fails. This time it's "No space left on device":
12:39:08 > Task :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker
12:39:08 java.io.IOException: No space left on device
12:39:08 com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/268/consoleFull

Is this specific to this PR? I'll confirm the word-count example in my local environment with my branch.

jenkins@apache-beam-jenkins-7:~$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       485G  484G 1001M 100% /

Probably not specific to your PR.

alexvanboxel · 2020-02-03T09:51:19Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2020-02-03T13:24:52Z

R: @lukecwik @kennknowles
21 successful checks! Would you review/merge this PR?

@alexvanboxel Thank you.

lukecwik · 2020-02-03T16:56:24Z

sdks/java/io/google-cloud-platform/build.gradle

@@ -41,9 +41,14 @@ dependencies {
  compile library.java.google_auth_library_credentials
  compile library.java.google_auth_library_oauth2_http
  compile library.java.google_cloud_bigquery_storage
-  compile library.java.google_cloud_bigtable_client_core
+  compile(library.java.google_cloud_bigtable_client_core) {


Why are these exclusions needed?

The maven pom that this module will generate has grpc-core and grpclb as top level dependencies and maven resolution rules will choose those over the ones specified by bigtable-client-core or google-cloud-core-grpc so I would have expected that we wouldn't need the exclusions.

These exclusions are for Maven's version range constraint. The dependency was transitively requesting io.grpc:grpc-core:jar:[1.26.0,1.26.0], while another dependency requests io.grpc:grpc-core:jar:[1.25.0,1.25.0]. This version range conflict would fail the build before Maven's dependency mediation (that would choose the closest one) happens. Detail in https://issues.apache.org/jira/browse/BEAM-8691?focusedCommentId=17025456&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17025456

I wasn't aware that the build would fail before the dependency mediation kicked in.

bigtable-client-core 1.13.0 and exclusion and gax

1387241

suztomo commented Jan 29, 2020

View reviewed changes

Controlling grpc-grpclb and grpc-core

8f97dba

suztomo force-pushed the bigtable-client-core branch from 478baf6 to 8f97dba Compare January 29, 2020 20:22

lukecwik reviewed Feb 3, 2020

View reviewed changes

suztomo requested a review from lukecwik February 3, 2020 20:09

lukecwik merged commit 7fa02bb into apache:master Feb 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-8691] bigtable-client-core dependency upgrade #10714

[BEAM-8691] bigtable-client-core dependency upgrade #10714

suztomo commented Jan 29, 2020 •

edited

suztomo Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

suztomo commented Jan 29, 2020

suztomo commented Jan 30, 2020

boyuanzz commented Jan 30, 2020

boyuanzz commented Jan 30, 2020

suztomo commented Jan 30, 2020

suztomo commented Jan 30, 2020

udim commented Jan 31, 2020

udim commented Jan 31, 2020

suztomo commented Jan 31, 2020

udim commented Jan 31, 2020

alexvanboxel commented Feb 3, 2020

suztomo commented Feb 3, 2020

lukecwik Feb 3, 2020

suztomo Feb 3, 2020 •

edited

lukecwik Feb 3, 2020

[BEAM-8691] bigtable-client-core dependency upgrade #10714

[BEAM-8691] bigtable-client-core dependency upgrade #10714

Conversation

suztomo commented Jan 29, 2020 • edited

Linkage Check result

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

suztomo Jan 29, 2020

Choose a reason for hiding this comment

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

lukecwik commented Jan 29, 2020

suztomo commented Jan 29, 2020

suztomo commented Jan 30, 2020

boyuanzz commented Jan 30, 2020

boyuanzz commented Jan 30, 2020

suztomo commented Jan 30, 2020

suztomo commented Jan 30, 2020

udim commented Jan 31, 2020

udim commented Jan 31, 2020

suztomo commented Jan 31, 2020

udim commented Jan 31, 2020

alexvanboxel commented Feb 3, 2020

suztomo commented Feb 3, 2020

lukecwik Feb 3, 2020

Choose a reason for hiding this comment

suztomo Feb 3, 2020 • edited

Choose a reason for hiding this comment

lukecwik Feb 3, 2020

Choose a reason for hiding this comment

suztomo commented Jan 29, 2020 •

edited

suztomo Feb 3, 2020 •

edited