Dependency conflicts with Apache Beam Cloud Dataflow runner #2890

kevmo314 · 2018-02-11T06:52:07Z

Hi, I'm running into the following exception when using Firestore:

java.lang.NoSuchMethodError: com.google.auth.oauth2.ServiceAccountCredentials.getProjectId()Ljava/lang/String;

	at com.google.firebase.FirebaseApp.getProjectId(FirebaseApp.java:307)
	at com.google.firebase.ImplFirebaseTrampolines.getProjectId(ImplFirebaseTrampolines.java:43)
	at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:32)
	at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:26)
	at com.google.firebase.cloud.FirestoreClient$FirestoreClientService.<init>(FirestoreClient.java:81)
	at com.google.firebase.cloud.FirestoreClient.getInstance(FirestoreClient.java:71)
	at com.google.firebase.cloud.FirestoreClient.getFirestore(FirestoreClient.java:64)
	at com.google.firebase.cloud.FirestoreClient.getFirestore(FirestoreClient.java:52)

My pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    ...
    <dependencies>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-jdk14</artifactId>
            <version>1.7.25</version>
        </dependency>
        <dependency>
            <groupId>com.intellij</groupId>
            <artifactId>annotations</artifactId>
            <version>12.0</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>24.0-jre</version>
        </dependency>
        <dependency>
            <groupId>com.google.auto.value</groupId>
            <artifactId>auto-value</artifactId>
            <version>1.5</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.assertj</groupId>
            <artifactId>assertj-core</artifactId>
            <version>3.9.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.0.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>5.0.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-all</artifactId>
            <version>1.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.11.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-core</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-direct-java</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
            <version>2.2.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.google.firebase</groupId>
            <artifactId>firebase-admin</artifactId>
            <version>5.8.0</version>
        </dependency>
    </dependencies>
</project>

and output of mvn dependency:tree

com.canigraduate.uchicago:scraper:jar:1.0-SNAPSHOT
+- org.slf4j:slf4j-jdk14:jar:1.7.25:compile
|  \- org.slf4j:slf4j-api:jar:1.7.25:compile
+- com.intellij:annotations:jar:12.0:compile
+- com.google.guava:guava:jar:24.0-jre:compile
|  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
|  +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
|  +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
|  +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
|  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
+- com.google.auto.value:auto-value:jar:1.5:provided
+- org.assertj:assertj-core:jar:3.9.0:test
+- org.junit.jupiter:junit-jupiter-engine:jar:5.0.3:test
|  +- org.apiguardian:apiguardian-api:jar:1.0.0:test
|  +- org.junit.platform:junit-platform-engine:jar:1.0.3:test
|  |  +- org.junit.platform:junit-platform-commons:jar:1.0.3:test
|  |  \- org.opentest4j:opentest4j:jar:1.0.0:test
|  \- org.junit.jupiter:junit-jupiter-api:jar:5.0.3:test
+- org.junit.jupiter:junit-jupiter-params:jar:5.0.3:test
+- org.hamcrest:hamcrest-all:jar:1.3:test
+- org.jsoup:jsoup:jar:1.11.2:compile
+- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
|  +- com.fasterxml.jackson.core:jackson-core:jar:2.8.9:compile
|  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9:compile
|  +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.9:compile
|  +- org.apache.avro:avro:jar:1.8.2:compile
|  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
|  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
|  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
|  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
|  |  \- org.tukaani:xz:jar:1.5:compile
|  +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
|  \- joda-time:joda-time:jar:2.4:compile
+- org.apache.beam:beam-runners-direct-java:jar:2.2.0:compile
+- org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
|  +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
|  |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
|  |  \- com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0:compile
|  +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
|  |  +- com.google.protobuf:protobuf-java:jar:3.2.0:compile
|  |  +- io.grpc:grpc-core:jar:1.2.0:compile
|  |  |  +- io.grpc:grpc-context:jar:1.2.0:compile
|  |  |  \- com.google.instrumentation:instrumentation-api:jar:0.3.0:compile
|  |  +- io.grpc:grpc-protobuf:jar:1.2.0:compile
|  |  |  +- com.google.protobuf:protobuf-java-util:jar:3.2.0:compile
|  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.2.0:compile
|  |  \- io.grpc:grpc-stub:jar:1.2.0:compile
|  +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
|  |  +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.2.0:compile
|  |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev355-1.22.0:compile
|  |  +- com.google.api:gax-grpc:jar:0.20.0:compile
|  |  |  +- com.google.api:gax:jar:1.3.1:compile
|  |  |  \- org.threeten:threetenbp:jar:1.3.3:compile
|  |  +- com.google.cloud:google-cloud-core-grpc:jar:1.2.0:compile
|  |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
|  |  +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18:compile
|  |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18:compile
|  |  |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.18:compile
|  |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
|  |  |  +- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
|  |  |  \- com.google.http-client:google-http-client-jackson:jar:1.20.0:compile
|  |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
|  |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.0:compile
|  |  +- io.grpc:grpc-auth:jar:1.2.0:compile
|  |  +- io.grpc:grpc-netty:jar:1.2.0:compile
|  |  |  +- io.netty:netty-codec-http2:jar:4.1.8.Final:compile (version selected from constraint [4.1.8.Final,4.1.8.Final])
|  |  |  \- io.netty:netty-handler-proxy:jar:4.1.8.Final:compile
|  |  |     \- io.netty:netty-codec-socks:jar:4.1.8.Final:compile
|  |  +- io.grpc:grpc-all:jar:1.2.0:compile
|  |  |  +- io.grpc:grpc-okhttp:jar:1.2.0:compile
|  |  |  |  +- com.squareup.okhttp:okhttp:jar:2.5.0:compile
|  |  |  |  \- com.squareup.okio:okio:jar:1.6.0:compile
|  |  |  \- io.grpc:grpc-protobuf-nano:jar:1.2.0:compile
|  |  |     \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
|  |  +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
|  |  +- com.google.cloud:google-cloud-spanner:jar:0.20.0-beta:compile
|  |  |  +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11:compile
|  |  |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
|  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11:compile
|  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11:compile
|  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
|  |  |  +- com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11:compile
|  |  |  |  \- com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11:compile
|  |  |  \- junit:junit:jar:4.12:compile
|  |  |     \- org.hamcrest:hamcrest-core:jar:1.3:compile
|  |  +- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile
|  |  +- com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0-pre3:compile
|  |  |  +- commons-logging:commons-logging:jar:1.2:compile
|  |  |  +- com.google.auth:google-auth-library-appengine:jar:0.7.0:compile
|  |  |  \- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
|  |  +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26:compile
|  |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9:compile
|  |  \- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
|  +- com.google.api-client:google-api-client:jar:1.22.0:compile
|  |  \- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
|  +- com.google.http-client:google-http-client:jar:1.22.0:compile
|  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
|  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
|  |     \- commons-codec:commons-codec:jar:1.3:compile
|  +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
|  +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev213-1.22.0:compile
|  +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
|  +- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
|  +- com.google.auth:google-auth-library-credentials:jar:0.7.1:compile
|  +- com.google.auth:google-auth-library-oauth2-http:jar:0.7.1:compile
|  \- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
|     +- com.google.api-client:google-api-client-java6:jar:1.20.0:compile
|     +- com.google.api-client:google-api-client-jackson2:jar:1.20.0:compile
|     \- com.google.oauth-client:google-oauth-client-java6:jar:1.20.0:compile
\- com.google.firebase:firebase-admin:jar:5.8.0:compile
   +- com.google.api-client:google-api-client-gson:jar:1.22.0:compile
   |  \- com.google.http-client:google-http-client-gson:jar:1.22.0:compile
   |     \- com.google.code.gson:gson:jar:2.1:compile
   +- com.google.api:api-common:jar:1.2.0:compile
   +- com.google.cloud:google-cloud-storage:jar:1.7.0:compile
   |  \- com.google.cloud:google-cloud-core-http:jar:1.7.0:compile
   |     \- com.google.http-client:google-http-client-appengine:jar:1.22.0:compile
   +- com.google.cloud:google-cloud-firestore:jar:0.25.0-beta:compile
   |  \- com.google.api.grpc:proto-google-cloud-firestore-v1beta1:jar:0.1.20:compile
   +- org.json:json:jar:20160810:compile
   +- io.netty:netty-codec-http:jar:4.1.14.Final:compile
   |  \- io.netty:netty-codec:jar:4.1.14.Final:compile
   +- io.netty:netty-handler:jar:4.1.14.Final:compile
   |  \- io.netty:netty-buffer:jar:4.1.14.Final:compile
   |     \- io.netty:netty-common:jar:4.1.14.Final:compile
   \- io.netty:netty-transport:jar:4.1.14.Final:compile
      \- io.netty:netty-resolver:jar:4.1.14.Final:compile

It looks like this is due to Beam pulling in old dependencies that aren't sufficient for firebase-admin, so I tried pinning

<dependency>
  <groupId>com.google.auth</groupId>
  <artifactId>google-auth-library-oauth2-http</artifactId>
  <version>0.9.0</version>
</dependency>

which resulted in #2496:

java.lang.NoClassDefFoundError: com/google/api/gax/rpc/ClientSettings

	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

so I tried adding

<dependency>
  <groupId>com.google.api</groupId>
  <artifactId>gax</artifactId>
  <version>1.19.0</version>
</dependency>
<dependency>
  <groupId>com.google.api</groupId>
  <artifactId>gax-grpc</artifactId>
  <version>1.19.0</version>
</dependency>

but this results in

java.lang.NoClassDefFoundError: com/google/api/gax/rpc/TransportProvider

	at com.google.cloud.firestore.FirestoreOptions.<clinit>(FirestoreOptions.java:41)
	at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:37)
	at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:26)
	at com.google.firebase.cloud.FirestoreClient$FirestoreClientService.<init>(FirestoreClient.java:81)
	at com.google.firebase.cloud.FirestoreClient.getInstance(FirestoreClient.java:71)

at which point I'm sort of stuck, since I thought this was provided by the gax-grpc artifact. I've also tried putting firebase-admin before all the beam dependencies, however this results in similar version conflicts...

Is there an easier way to resolve the dependency conflict between Beam and firebase-admin?

The text was updated successfully, but these errors were encountered:

kevmo314 · 2018-02-11T07:12:00Z

For posterity, if I put firebase-admin first, then I end up with a conflict on io.grpc:grpc-core as Beam seems to require 1.5.0. Pinning that to 1.9.0 as requested by firebase-admin results in

Feb 11, 2018 2:09:55 AM com.google.api.gax.core.PropertiesProvider logMissingProperties
WARNING: Warning: Failed to open properties resource at '%s' of the given class '%s'

Feb 11, 2018 2:09:55 AM com.google.api.gax.core.PropertiesProvider logMissingProperties
WARNING: Warning: Failed to open properties resource at '%s' of the given class '%s'

Feb 11, 2018 2:09:56 AM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved run
WARNING: [io.grpc.internal.ManagedChannelImpl-1] Unexpected exception from LoadBalancer
java.lang.AbstractMethodError: io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(Ljava/net/SocketAddress;Ljava/lang/String;Ljava/lang/String;Lio/grpc/internal/ProxyParameters;)Lio/grpc/internal/ConnectionClientTransport;
	at io.grpc.internal.CallCredentialsApplyingTransportFactory.newClientTransport(CallCredentialsApplyingTransportFactory.java:49)
	at io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:211)
	at io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:190)
	at io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1020)
	at io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
	at io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:924)
	at io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:72)
	at io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:892)
	at io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:936)
	at io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Thrown from handleResolvedAddresses(): java.lang.AbstractMethodError: io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(Ljava/net/SocketAddress;Ljava/lang/String;Ljava/lang/String;Lio/grpc/internal/ProxyParameters;)Lio/grpc/internal/ConnectionClientTransport;

	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:528)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:85)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)
	at com.canigraduate.uchicago.pipeline.firestore.FirestoreServiceTest.getFirestore(FirestoreServiceTest.java:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Thrown from handleResolvedAddresses(): java.lang.AbstractMethodError: io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(Ljava/net/SocketAddress;Ljava/lang/String;Ljava/lang/String;Lio/grpc/internal/ProxyParameters;)Lio/grpc/internal/ConnectionClientTransport;
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.createException(GrpcApiExceptionFactory.java:88)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:112)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1341)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1027)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:868)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:713)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:458)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:433)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:392)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Thrown from handleResolvedAddresses(): java.lang.AbstractMethodError: io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(Ljava/net/SocketAddress;Ljava/lang/String;Ljava/lang/String;Lio/grpc/internal/ProxyParameters;)Lio/grpc/internal/ConnectionClientTransport;
	at io.grpc.Status.asRuntimeException(Status.java:526)
	... 19 more
Caused by: java.lang.AbstractMethodError: io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(Ljava/net/SocketAddress;Ljava/lang/String;Ljava/lang/String;Lio/grpc/internal/ProxyParameters;)Lio/grpc/internal/ConnectionClientTransport;
	at io.grpc.internal.CallCredentialsApplyingTransportFactory.newClientTransport(CallCredentialsApplyingTransportFactory.java:49)
	at io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:211)
	at io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:190)
	at io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1020)
	at io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
	at io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:924)
	at io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:72)
	at io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:892)
	at io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:936)
	at io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
	... 3 more

PandaMonkey · 2018-02-11T15:08:07Z

Hi, @kevmo314 , I have carefully reviewed your issue and spend some energy to analyze your dependency tree.
For the first issue "java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: com.google.auth.oauth2.ServiceAccountCredentials.getProjectId()",
it caused by the duplicate library "com.google.auth:google-auth-library-oauth2-http". Your project introduced three versions of this library, i.e.,
0.7.1, 0.7.0, and 0.8.0. Of them, 0.7.1 is the acturally loaded one and 0.8.0 is the correct version really required by your code.

When you tried to solve the problem at first time, a new version 0.9.0 was forcedly introduced, which exacerbated the problem.
So maybe the right way to work around this problem is that adding the following patch in your original pom file:

	<dependency>
		<groupId>com.google.api</groupId>
		<artifactId>google-auth-library-oauth2-http</artifactId>
		<version>0.8.0</version>
	</dependency>

For the second problem "java.lang.NoClassDefFoundError: com/google/api/gax/rpc/ClientSettings：",
You directly introduced com.google.api:gax 1.19.0 (pretty new for your project), however, the older version exists in the dependency tree,
as a result, maybe com.google.api:gax 1.9.0 is a better choice. ^_^

You can add the following patch in your pom.xml:

	<dependency>
		<groupId>com.google.api</groupId>
		<artifactId>gax</artifactId>
		<version>1.9.0</version>
	</dependency>

Then the problem may be disappeared. Hope my answer can help you.

kevmo314 · 2018-02-11T16:38:32Z

I tried pinning to the versions you suggested and received the same exception

java.lang.NoClassDefFoundError: com/google/api/gax/grpc/GrpcTransport

	at com.google.cloud.firestore.FirestoreOptions$DefaultFirestoreRpcFactory.create(FirestoreOptions.java:71)
	at com.google.cloud.firestore.FirestoreOptions$DefaultFirestoreRpcFactory.create(FirestoreOptions.java:63)
	at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:435)
	at com.google.cloud.firestore.FirestoreOptions.getFirestoreRpc(FirestoreOptions.java:163)
	at com.google.cloud.firestore.FirestoreImpl.<init>(FirestoreImpl.java:140)

I don't think pinning minor versions would solve the issue either, as minor versions are supposed to be backwards-compatible with semantic versioning. If they're not, then that suggests a more systemic problem in the Google Java APIs...

kevmo314 · 2018-02-11T17:10:52Z

Okay, so it turns out that there are substantial differences between 1.9.0 and higher versions.

That's insane, that completely goes against the point of using semantic versioning. I'm going to echo the sentiments in #2889, why is it so hard to get these APIs functional?

And my original question still stands: Is there a sane way to resolve these dependencies?

PandaMonkey · 2018-02-12T04:18:01Z

@kevmo314 , Sorry, my mistake.
The missed class com/google/api/gax/grpc/GrpcTransport is in com.google.api:gax-grpc 0.25.1. This library is not be loaded. You can upgrade com.google.cloud:google-cloud-firestore:0.25.0-beta to 0.34.0-beta. In this way, the higher version does not need this missing class.
So adding this patch may work:

            <dependency>
		<groupId>com.google.cloud</groupId>
		<artifactId>google-cloud-firestore</artifactId>
		<version>0.34.0-beta</version>
	</dependency>

Thanks.

kevmo314 · 2018-02-12T04:34:56Z

Thanks for the help, but that just leads me to yet another exception:

java.lang.NoClassDefFoundError: com/google/api/gax/rpc/StubSettings

	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

I also had to refactor some of my code because apparently the API around collections is different between 0.25.0-beta and 0.34.0-beta, which is certifiably crazy.

pongad · 2018-02-12T04:51:20Z

@PandaMonkey Thank you for the analysis!

@garrettjonesgoogle My understanding: we are (were?) using unstable surfaces to implement stable ones. I believe this works OK if the two surfaces are in the same Maven package. In this case, they're not. BEAM depends on Spanner 0.20.0-beta, which depends on an older GAX. Now @kevmo314 is trying to use Firestore which wants new GAX.

vkedia · 2018-02-12T18:10:37Z

cc @mairbek
We actually want to upgrade Beam to use the latest version of spanner as well. But Beam also depends on really old versions of grpc and possibly some other things like gax etc( irrespective of spanner). So it has been a challenge to do so.
We might want to release shaded artifacts of Spanner and Firestore so that they can be used with Beam at arbitrary versions.

kevmo314 · 2018-02-12T18:48:54Z

Is there a short term workaround (eg can I force maven to shade Firestore or Beam)? I'm blocked at the moment due to this incompatibility, but currently my only practical option seems to be to remove the firebase-admin dependency and manually issue GRPC calls. Or I guess manually find versions that are compatible with both. The former seems less painful, actually. :)

pongad · 2018-02-15T03:22:13Z

@kevmo314 I'm not sure there's a version of gax that's compatible with both. The oldest version of Firestore I can see depends on gax 1.9.0.

@schmidt-sebastian Do you have any thoughts?

garrettjonesgoogle · 2018-02-19T05:55:15Z

Hi everyone, I think it would be good to set expectations correctly around versioning.

Spanner and Firestore are both 0.x versions and accordingly are marked beta, so you will have to expect some breaking changes when using them, especially together.
In order to promise stability as early as possible for basic cases, we had to do a "partial stability" approach. This means that core things in gax don't break between minor versions, but at the same time there are also components in gax that break between versions; however, they are all marked with annotations to indicate this (e.g. with @BetaApi).
The only way to manage the complexity of a large number of stable & unstable APIs is to provide releases that include everything, which we do. That means for any set of APIs, there should be a version of each that is compatible, since they were released at the same time.
Trying to manually fiddle with versions across multiple packages is a path to madness. I highly recommend using google-cloud-bom (see https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bom ) instead to set your versions in a way that makes them all compatible.

kevmo314 · 2018-02-19T10:18:19Z

Thanks @garrettjonesgoogle for the tip about google-cloud-bom, I'll give that a shot and see if that works.

Spanner and Firestore are both 0.x versions and accordingly are marked beta, so you will have to expect some breaking changes when using them, especially together.

As @pongad mentioned, I'm using neither Spanner nor Firestore directly, but rather I'm depending on Apache Beam and firebase-admin, both of which should be "stable" as they have versions >=1.0.0. I've come to realize that firebase-admin is nothing more than a thin wrapper around the 0.x google-cloud-firestore, so perhaps that's the real issue.

kevmo314 · 2018-02-19T13:10:25Z

It looks like google-cloud-bom doesn't pin the version for io.grpc:grpc-core, so Maven still complains about not being able to resolve the version correctly. :(

garrettjonesgoogle · 2018-02-20T14:47:20Z

Ahh, I think grpc-core slipped through the cracks - we never had an explicit dependency on grpc-core because the other grpc packages pull it in, so it didn't make it into the bom either. We should add it in. You can set the version of that package manually though, so that it uses 1.9.0 like the rest.

In my scan through this issue I didn't notice before that firebase-admin was a package outside of google-cloud-java. It's definitely a problem that it is >= 1.0.0 and depends on the 0.x google-cloud-firestore. Unlike google-cloud-java & gax which step with each other, firebase-admin is managed completely independently.

If you have to ditch firebase-admin, there is a halfway solution that's easier than direct grpc - you can directly use google-cloud-firestore, which is easier to keep consistent with google-cloud-spanner than firebase-admin, but is more usable than direct grpc stubs.

kevmo314 · 2018-02-26T18:48:26Z

Just to follow up with this, I ended up giving up and using the Firestore REST API, as it was easier to debug than the GRPC API just due to familiarity.

I did try the suggestion to use google-cloud-firestore directly along with google-cloud-bom, however I ran into another NoClassDefFound exception even after pinning grpc-core and I was too frustrated to continue. I'd be happy to help with any additional output or provide a sample project for further debugging, but I guess until both Spanner and Firestore have stabilized, finding a solution here is too painful.

pongad · 2018-02-27T02:24:33Z

@kevmo314 I'm sorry to hear that. Upgrading BEAM's dependencies (apache/beam#4707) should help in the long run though.

kevmo314 · 2018-05-20T07:06:54Z

Any update on this? Both apache/beam#4707 and apache/beam#4727 seem to have stalled.

As much as I enjoy spending 50% more on Dataflow to negotiate HTTPS requests over and over again, it would be nice to uh, not.

garrettjonesgoogle · 2018-05-21T04:38:49Z

We haven't been pursuing anything on google-cloud-java's side - this seemed to be more in BEAM's court since they're the ones with versions that are so far behind (grpc 1.2 is > 1 year old now).

kevmo314 · 2018-05-23T06:46:50Z

Ah, sounds good, I'll check in with them, thanks!

garrettjonesgoogle · 2018-07-19T18:14:33Z

apache/beam#5988 will bring Beam up to date, and it has passed tests.

chingor13 · 2018-10-09T19:38:34Z

Looks like this is fixed.

pongad added type: question Request for information or clarification. Not an issue. dependencies labels Feb 12, 2018

pongad mentioned this issue Feb 12, 2018

java.lang.ClassNotFoundException: com.google.api.gax.rpc.StubSettings #2889

Closed

pongad mentioned this issue Feb 19, 2018

[BEAM-3321] update google-cloud-spanner dependency apache/beam#4707

Closed

This was referenced Feb 20, 2018

Add grpc-core to google-cloud-bom #2924

Closed

firebase-admin exposes unstable APIs firebase/firebase-admin-java#144

Closed

kevmo314 mentioned this issue May 23, 2018

[BEAM-3519] Overrides gRPC dependency for google-cloud-platform and upgrades protobuf apache/beam#4727

Closed

yihanzhen added the status: blocked Resolving the issue is dependent on other work. label Jun 6, 2018

chingor13 closed this as completed Oct 9, 2018

JustinBeckwith assigned chingor13 Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dependency conflicts with Apache Beam Cloud Dataflow runner #2890

Dependency conflicts with Apache Beam Cloud Dataflow runner #2890

kevmo314 commented Feb 11, 2018 •

edited

Loading

kevmo314 commented Feb 11, 2018

PandaMonkey commented Feb 11, 2018 •

edited

Loading

kevmo314 commented Feb 11, 2018 •

edited

Loading

kevmo314 commented Feb 11, 2018

PandaMonkey commented Feb 12, 2018 •

edited

Loading

kevmo314 commented Feb 12, 2018

pongad commented Feb 12, 2018

vkedia commented Feb 12, 2018

kevmo314 commented Feb 12, 2018

pongad commented Feb 15, 2018

garrettjonesgoogle commented Feb 19, 2018

kevmo314 commented Feb 19, 2018

kevmo314 commented Feb 19, 2018

garrettjonesgoogle commented Feb 20, 2018

kevmo314 commented Feb 26, 2018 •

edited

Loading

pongad commented Feb 27, 2018

kevmo314 commented May 20, 2018

garrettjonesgoogle commented May 21, 2018

kevmo314 commented May 23, 2018

garrettjonesgoogle commented Jul 19, 2018

chingor13 commented Oct 9, 2018

Dependency conflicts with Apache Beam Cloud Dataflow runner #2890

Dependency conflicts with Apache Beam Cloud Dataflow runner #2890

Comments

kevmo314 commented Feb 11, 2018 • edited Loading

kevmo314 commented Feb 11, 2018

PandaMonkey commented Feb 11, 2018 • edited Loading

kevmo314 commented Feb 11, 2018 • edited Loading

kevmo314 commented Feb 11, 2018

PandaMonkey commented Feb 12, 2018 • edited Loading

kevmo314 commented Feb 12, 2018

pongad commented Feb 12, 2018

vkedia commented Feb 12, 2018

kevmo314 commented Feb 12, 2018

pongad commented Feb 15, 2018

garrettjonesgoogle commented Feb 19, 2018

kevmo314 commented Feb 19, 2018

kevmo314 commented Feb 19, 2018

garrettjonesgoogle commented Feb 20, 2018

kevmo314 commented Feb 26, 2018 • edited Loading

pongad commented Feb 27, 2018

kevmo314 commented May 20, 2018

garrettjonesgoogle commented May 21, 2018

kevmo314 commented May 23, 2018

garrettjonesgoogle commented Jul 19, 2018

chingor13 commented Oct 9, 2018

kevmo314 commented Feb 11, 2018 •

edited

Loading

PandaMonkey commented Feb 11, 2018 •

edited

Loading

kevmo314 commented Feb 11, 2018 •

edited

Loading

PandaMonkey commented Feb 12, 2018 •

edited

Loading

kevmo314 commented Feb 26, 2018 •

edited

Loading