It is trying to connect 127.0.0.1:5567?

[karayakar]

Problem encountered on https://dotnet.microsoft.com/learn/data/spark-tutorial/code
Operating System: windows

Provide details about the problem you are experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (10061): No connection could be made because the target machine actively refused it. 127.0.0.1:5567
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at Microsoft.Spark.Network.DefaultSocketWrapper.Connect(IPAddress remoteaddr, Int32 port, String secret)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.GetConnection()
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String methodName, Object[] args)
   at Microsoft.Spark.Sql.Builder..ctor()
   at Microsoft.Spark.Sql.SparkSession.Builder()

[imback82]

5567 port is used for debugging. Can you provide more details such as spark-submit command, etc.?

[trongnguyenbinh]

I ran into the same issue, it fine when I run this command

spark-submit --class org.apache.spark.deploy.DotnetRunner --master local bin\Debug\netcoreapp2.2\microsoft-spark-2.4.x-0.3.0.jar dotnet bin\Debug\netcoreapp2.2\MySparkApp.dll

but throw exception when I tried to debug.

System.Net.Internals.SocketExceptionFactory.ExtendedSocketException
  HResult=0x80004005
  Message=No connection could be made because the target machine actively refused it 127.0.0.1:5567
  Source=System.Net.Sockets
  StackTrace:
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at Microsoft.Spark.Network.DefaultSocketWrapper.Connect(IPAddress remoteaddr, Int32 port, String secret)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.GetConnection()
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String methodName, Object[] args)
   at Microsoft.Spark.Sql.Builder..ctor()
   at Microsoft.Spark.Sql.SparkSession.Builder()
   at mySparkApp.Program.Main(String[] args) in E:\Develop\mySparkApp\Program.cs:line 11

[imback82]

Please follow this doc: https://github.com/dotnet/spark/blob/master/docs/developer-guide.md#debugging-spark-net-application

[trongnguyenbinh]

Great! thanks @imback82

[rapoth]

@trongnguyenbinh: I'm closing this issue. Please feel free to create another issue if you get blocked.

[Devtr0n]

When following the instructions for "Debugging Spark .NET application"....

Where is the <path-to-microsoft-spark-jar> ??? I don't see any .jar files. Help?

I have "C:\bin\Microsoft.Spark.Worker-2.1.1" but no .jar files :(