java.lang.NoClassDefFoundError: org/spark-project/jetty/alpn/ALPN$Provider when instantiating MetricServiceClient

[Berg3]

When trying to instantiate a MetricServiceClient in order to perform Google Stackdriver Monitoring using the following code in a DataProc cluster image 1.0:

MetricServiceSettings settings = MetricServiceSettings.defaultBuilder()
                .setCredentialsProvider(credentialsProvider)
                .build();
MetricServiceClient metricServiceClient =  MetricServiceClient.create(settings);

I get the following stack trace:

Exception in thread "main" java.lang.NoClassDefFoundError: org/spark-project/jetty/alpn/ALPN$Provider
	at io.netty.handler.ssl.JdkAlpnApplicationProtocolNegotiator$1.<init>(JdkAlpnApplicationProtocolNegotiator.java:26)
	at io.netty.handler.ssl.JdkAlpnApplicationProtocolNegotiator.<clinit>(JdkAlpnApplicationProtocolNegotiator.java:24)
	at io.netty.handler.ssl.JdkSslContext.toNegotiator(JdkSslContext.java:237)
	at io.netty.handler.ssl.JdkSslClientContext.<init>(JdkSslClientContext.java:189)
	at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:729)
	at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:223)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:470)
	at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:338)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:305)
	at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:125)
	at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:110)
	at com.google.api.gax.grpc.GrpcTransportProvider.getTransport(GrpcTransportProvider.java:98)
	at com.google.api.gax.grpc.GrpcTransportProvider.getTransport(GrpcTransportProvider.java:59)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:97)
	at com.google.cloud.monitoring.v3.stub.GrpcMetricServiceStub.create(GrpcMetricServiceStub.java:161)
	at com.google.cloud.monitoring.v3.MetricServiceSettings.createStub(MetricServiceSettings.java:195)
	at com.google.cloud.monitoring.v3.MetricServiceClient.<init>(MetricServiceClient.java:140)
	at com.google.cloud.monitoring.v3.MetricServiceClient.create(MetricServiceClient.java:122)

This is the dependency in the pom.xml file (spark version 1.6.3):

  <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-monitoring</artifactId>
      <version>0.22.0-alpha</version>
      <exclusions>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.10</artifactId>
      <version>${spark.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-client</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.commons</groupId>
          <artifactId>commons-lang3</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.java.dev.jets3t</groupId>
          <artifactId>jets3t</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
      </exclusions>
      <!--<scope>provided</scope>-->
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-catalyst_2.10</artifactId>
      <version>${spark.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-client</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.commons</groupId>
          <artifactId>commons-lang3</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.java.dev.jets3t</groupId>
          <artifactId>jets3t</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
      </exclusions>
      <!--<scope>provided</scope>-->
    </dependency>

    <dependency>
      <groupId>com.databricks</groupId>
      <artifactId>spark-csv_2.11</artifactId>
      <version>${spark.csv.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-client</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.commons</groupId>
          <artifactId>commons-lang3</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.java.dev.jets3t</groupId>
          <artifactId>jets3t</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

Any advice as to how to resolve this issue would be appreciated. Thanks.

Edit: I have no issue when running my program locally, the issue only appears when running on the Dataproc cluster.

[michaelbausor]

Hi @Berg3, a couple of things you could try:

• Upgrade to the latest release (0.23.1) and confirm you are still seeing the same issue
• Have a look at Error: Jetty ALPN/NPN has not been properly configured. #2266 - it doesn't seem like exactly the same issue, but it does relate to Jetty.
• Let us know what platform you are running on

[michaelbausor]

Something else you could try is to check and compare what version of netty is installed locally and on the Dataproc cluster. You can use the command mvn dependency:tree to get a list of dependencies for your project. You can see only the netty dependencies with mvn dependency:tree | grep io.netty

If you find a difference in the netty version, you make want to try explicitly specifying dependencies on relevant io.netty packages in your pom to make sure the correct version is being used.

[Berg3]

As I did some more digging, I simplified what I was trying to do to try to isolate the issue since my program was running in a much larger established codebase, and it turns out that I am having I think essentially the same issue as was in the post @michaelbausor mentioned: #2266 So I looked at the post and on the dataproc cluster I followed the steps here: https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-util/google-cloud-compat-checker and I get the following output:

OS details:
  os.detected.name: linux
  os.detected.arch: x86_64
  os.detected.classifier: linux-x86_64
  os.detected.release: debian
  os.detected.release.version: 8
JVM details:
  Java version: 1.8.0_131
  Java specification version: 1.8
  JVM bit mode: 64
OpenSSL details:
  open ssl is available: true
  ALPN is supported: true
Checking compatibility...
  [PASS] This OS + architecture is supported.
  [PASS] 64-bit JVM is supported.
  [PASS] Open SSL is available
  [PASS] Open SSL ALPN is supported
Result: UNKNOWN (checker implementation not complete)
  Based on what was checked, nothing was identified that would
  prevent you from using grpc-based APIs.

but when running the simplified program I still get the error:

Exception in thread "main" java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
	at io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:159)
	at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:136)
	at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:124)
	at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:94)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(NettyChannelBuilder.java:525)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(NettyChannelBuilder.java:518)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:457)
	at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:326)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:315)
	at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:131)
	at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:116)
	at com.google.api.gax.grpc.GrpcTransportProvider.getTransport(GrpcTransportProvider.java:98)
	at com.google.api.gax.grpc.GrpcTransportProvider.getTransport(GrpcTransportProvider.java:59)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:97)
	at com.google.cloud.monitoring.v3.stub.GrpcMetricServiceStub.create(GrpcMetricServiceStub.java:161)
	at com.google.cloud.monitoring.v3.MetricServiceSettings.createStub(MetricServiceSettings.java:195)
	at com.google.cloud.monitoring.v3.MetricServiceClient.<init>(MetricServiceClient.java:159)
	at com.google.cloud.monitoring.v3.MetricServiceClient.create(MetricServiceClient.java:141)

from running the command:

MetricServiceSettings settings = MetricServiceSettings.newBuilder()
                .setCredentialsProvider(credentialsProvider)
                .build();

MetricServiceClient metricServiceClient =  MetricServiceClient.create(settings);

where the create(settings) command causes the error

This is the output I get when running the command:

mvn dependency:tree | grep io.netty

[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:2.0.3.Final:compile
[INFO] |  |  +- io.netty:netty-codec-http2:jar:4.1.14.Final:compile (version selected from constraint [4.1.14.Final,4.1.14.Final])
[INFO] |  |  |  +- io.netty:netty-codec-http:jar:4.1.14.Final:compile
[INFO] |  |  |  |  \- io.netty:netty-codec:jar:4.1.14.Final:compile
[INFO] |  |  |  \- io.netty:netty-handler:jar:4.1.14.Final:compile
[INFO] |  |  |     \- io.netty:netty-buffer:jar:4.1.14.Final:compile
[INFO] |  |  |        \- io.netty:netty-common:jar:4.1.14.Final:compile
[INFO] |  |  \- io.netty:netty-handler-proxy:jar:4.1.14.Final:compile
[INFO] |  |     +- io.netty:netty-transport:jar:4.1.14.Final:compile
[INFO] |  |     |  \- io.netty:netty-resolver:jar:4.1.14.Final:compile
[INFO] |  |     \- io.netty:netty-codec-socks:jar:4.1.14.Final:compile
[INFO] |  +- io.netty:netty-all:jar:4.0.43.Final:compile
[INFO] |  +- io.netty:netty:jar:3.9.9.Final:compile

[Berg3]

After doing a little more digging, I decided to make sure that OpenSSL is available directly in my project, so I added the following check:

System.out.println("OpenSsl.isAvailable: " + OpenSsl.isAvailable());
if (!OpenSsl.isAvailable()) {
    OpenSsl.unavailabilityCause().printStackTrace();
}
System.out.println("OpenSsl.isAlpnSupported: " + OpenSsl.isAlpnSupported());

directly before I instantiate the metric service client. Locally, I get the following print statements printed to console:

OpenSsl.isAvailable: true
OpenSsl.isAlpnSupported: true

however, despite passing the google-cloud-compat-checker as I mentioned in the previous comment, when I run my program on the dataproc cluster, I get the following error:

OpenSsl.isAvailable: false
java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:87)
	at com.apigee.analytics.platform.StackdriverController.sendUapStatsToStackdriver(StackdriverController.java:121)
	at com.apigee.analytics.platform.StackdriverController.main(StackdriverController.java:90)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
OpenSsl.isAlpnSupported: false

I probably should mention that locally, I am running via IntelliJ whereas on the dataproc cluster I am running the class through a spark-submit job

[Berg3]

The issue was because I was running the job as a spark-submit job without specifying:

--conf spark.driver.userClassPathFirst=true \
--conf spark.executor.userClassPathFirst=true \

so the wrong io.netty jar files were used