Using with .NET Interactive notebooks locally

[WhitWaldo]

I'd ideally like to submit jobs to my Spark cluster (local or remote) via a Jupyter notebook. It turns out that there's a recent update to .NET Interactive to allow me to use their C# REPL within Jupyter, but I'm unable to find any documentation indicating how to use this with Spark.NET.

Both Synapse and Databricks offer support to use notebooks online with Spark, so I'm looking for guidance on how to do the same locally. Thank you!

[imback82]

Yes, this is doable.

First, run a spark-submit with a debug mode:

> spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner microsoft-spark-2.4.x-0.10.0.jar debug
20/05/22 21:01:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/05/22 21:01:06 INFO DotnetRunner: Starting DotnetBackend with .
20/05/22 21:01:06 INFO DotnetRunner: Port number used by DotnetBackend is 5567
***********************************************************************
* .NET Backend running debug mode. Press enter to exit *
***********************************************************************

Now, open up the Jupyter notebook with dotnet-interactive kernel (C#), install Microsoft.Spark Nuget with #r "nuget: Microsoft.Spark", and start writing your application:

The ability to write UDF in the notebook will be introduced with #515 and #517 and once they are both merged, we will write up a guide on how to use this locally.

Thanks!

[imback82]

cc: @jonsequitur (just FYI)

[Niharikadutta]

@WhitWaldo We believe this issue to be resolved, do you need any further help in running your spark jobs via Jupyter?

[Niharikadutta]

Hi, we are going to close this issue as it has been inactive for a while. Please feel free to re-open it if the issue persists and/or there are any new updates. Thank you!

[imback82]

Reopening to follow up on the documentation.

[Niharikadutta]

@WhitWaldo We recently wrote an article to document the steps to use .NET interactive notebooks locally using .NET for Apache Spark, please feel free to check it out here.
Closing this issue.