Netty incompatibility when using Pub/Sub with Dataproc

[nsphung]

I'm trying to send messages from Spark Google Dataproc to Google PubSub. But I'm stuck with the following errors when trying to initialize the PubSub client:

17/01/10 15:12:28 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, fca-cluster-dev-w-1.c.fcms-sandbox-gce.internal): java.lang.NoSuchMethodError: io.netty.util.AttributeKey.valueOf(Ljava/lang/Class;Ljava/lang/String;)Lio/netty/util/AttributeKey;
    at io.grpc.netty.Utils.<clinit>(Utils.java:85)
    at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:311)
    at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:280)
    at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:230)
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:239)
    at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:119)
    at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:106)
    at com.google.api.gax.grpc.ProviderManager.getChannel(ProviderManager.java:106)
    at com.google.api.gax.grpc.ChannelAndExecutor.create(ChannelAndExecutor.java:67)
    at com.google.api.gax.grpc.ClientSettings.getChannelAndExecutor(ClientSettings.java:78)
    at com.google.cloud.pubsub.spi.v1.PublisherClient.<init>(PublisherClient.java:182)
    at com.google.cloud.pubsub.spi.v1.PublisherClient.create(PublisherClient.java:173)
    at com.google.cloud.pubsub.spi.DefaultPubSubRpc.<init>(DefaultPubSubRpc.java:168)
    at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:69)
    at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:63)
    at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:482)
    at com.google.cloud.pubsub.PubSubImpl.<init>(PubSubImpl.java:115)
    at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:44)
    at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:39)
    at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:469)
    at services.PubSubService$.<init>(PubSubService.scala:14)
    at services.PubSubService$.<clinit>(PubSubService.scala)
    at Main$$anonfun$createContext$5$$anonfun$apply$1.apply(Main.scala:93)
    at Main$$anonfun$createContext$5$$anonfun$apply$1.apply(Main.scala:92)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$27.apply(RDD.scala:894)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$27.apply(RDD.scala:894)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

The client initialization is as follow:

 val pubsub: PubSub = PubSubOptions.getDefaultInstance().getService()

I look around and it's an issue between the netty version of Spark Google Dataproc netty-all-4.0.29.Final vs the one on google-cloud-pubsub (com.google.cloud) library in netty-all-4.1.3.Final. Did anyone manage to resolve this ?

Same issue as #1331 but with Google Dataproc (Spark).

Regards,

[lesv]

Dataproc and Dataflow should be considered important use cases of Pub/Sub. They may, however, have their own way of doing things. (at least Dataflow)

[garrettjonesgoogle]

The resolution for this issue may be the same as for #1331 (shading netty).