Caused by: java.nio.channels.UnresolvedAddressException

[jacace]

It fails when running in with docker compose with two different error messages:

Connecting to master spark://localhost:7077...
Failed to connect to master localhost:7077

• If using master spark://spark-master:7077 the connection fails without further explanation on either side.
• But if using spark://localhost:7077 the connection fails too in the master side with the error below, which makes me think that it is either a mismatch with network libraries or another protocol related error.

spark-master | 21/02/09 20:15:57 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 8684905230527837511
spark-master | java.io.EOFException
spark-master | at java.io.DataInputStream.readFully(DataInputStream.java:197)
spark-master | at java.io.DataInputStream.readUTF(DataInputStream.java:609)
spark-master | at java.io.DataInputStream.readUTF(DataInputStream.java:564)
spark-master | at org.apache.spark.rpc.netty.RequestMessage$.readRpcAddress(NettyRpcEnv.scala:593)
spark-master | at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:603)
spark-master | at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:662)
spark-master | at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:647)
spark-master | at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:180)
spark-master | at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:103)
spark-master | at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
spark-master | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
spark-master | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
spark-master | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
spark-master | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
spark-master | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
spark-master | at java.lang.Thread.run(Thread.java:748)
spark-master | 21/02/09 20:16:17 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 8857927301048500268
spark-master | java.io.EOFException
spark-master | at java.io.DataInputStream.readFully(DataInputStream.java:197)
spark-master | at java.io.DataInputStream.readUTF(DataInputStream.java:609)
spark-master | at java.io.DataInputStream.readUTF(DataInputStream.java:564)
spark-master | at org.apache.spark.rpc.netty.RequestMessage$.readRpcAddress(NettyRpcEnv.scala:593)
spark-master | at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:603)
spark-master | at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:662)
spark-master | at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:647)
spark-master | at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:180)
spark-master | at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:103)
spark-master | at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
spark-master | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
spark-master | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
spark-master | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
spark-master | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
spark-master | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
spark-master | at java.lang.Thread.run(Thread.java:748)
spark-master | 21/02/09 20:16:37 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 8019921504629488561
spark-master | java.io.EOFException
spark-master | at java.io.DataInputStream.readFully(DataInputStream.java:197)
spark-master | at java.io.DataInputStream.readUTF(DataInputStream.java:609)
spark-master | at java.io.DataInputStream.readUTF(DataInputStream.java:564)
spark-master | at org.apache.spark.rpc.netty.RequestMessage$.readRpcAddress(NettyRpcEnv.scala:593)
spark-master | at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:603)
spark-master | at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:662)
spark-master | at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:647)
spark-master | at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:180)
spark-master | at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:103)
spark-master | at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
spark-master | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
spark-master | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
spark-master | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
spark-master | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
spark-master | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
spark-master | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
spark-master | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
spark-master | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
spark-master | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
spark-master | at java.lang.Thread.run(Thread.java:748)
spark-master | 21/02/09 20:16:57 INFO Master: 172.19.0.1:60216 got disassociated, removing it.

[jacace]

It seems to be related with a version mismatch as per below:
https://stackoverflow.com/questions/63071575/unable-to-connect-apache-spark-running-in-docker

[jacace]

Issues fixed with teh following newer libraries which depend on Scala 2.11:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_**2.11**</artifactId>
  <version>2.4.7</version>
  <scope>provided</scope>
</dependency>    

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-kafka-0-10_**2.11**</artifactId>
  <version>2.4.7</version>
</dependency>

[jacace]

As with any Spark applications, spark-submit is used to launch your application.

For Scala and Java applications, if you are using SBT or Maven for project management, then package spark-streaming-kafka-0-10_2.11 and its dependencies into the application JAR. Make sure spark-core_2.11 and spark-streaming_2.11 are marked as provided dependencies as those are already present in a Spark installation.

(extracted from: https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html)