spark 3.x support

[antonkulaga]

Spark 3.x is released ( https://spark.apache.org/releases/spark-release-3-0-0.html ), would be cool to have it in polynote

[fivejjs]

This is a desirable feature.

[aaronzo]

@antonkulaga What exactly doesn't work for you? I've just installed spark 3 and used the latest polynote-2.12.tar.gz release which uses scala 2.12, the same version that spark 3.0.0 uses, and All is running swimmingly so far.

My setup is just a local dev setup though, with spark running with --master local[*] on WSL in Windows. I have not tested it on any meaningfully sized spark clusters yet.

For the record, I'm also using GraalVM with Java 11, though I doubt that makes any difference

[hohonuuli]

Spark 3 in polynote 0.3.11 seems to work just fine. You can try it out in Docker:

docker run -p 8192:8192 -v $HOME/Documents/Notebooks:/opt/polynote/notebooks --name polynote mbari/polynote:0.3.11

Be sure to change the path of the volume $HOME/Documents/Notebooks to the path of a directory on your local computer.

[marrocksd]

The kernel cant start if I try to add some spark properties in UI

[antonkulaga]

I am having issues like:

polynote.kernel.remote.SocketTransport$ProcessDied$: Kernel died unexpectedly

Uncaught exception: Kernel died unexpectedly (polynote.kernel.remote.SocketTransport$ProcessDied$)

[antonkulaga]

I managed to sovle kernel-died error, but now reading csv-s do not work. They run for ages without providing the results

[antonkulaga]

A simple code like:

val df = spark.read.format("csv").option("header", true).option("sep", "\t").load("/data/ensembl/99/ensembl_anage_vertebrates.tsv")
df.printSchema()

runs for ages in polynote for spark 3

[aaronzo]

As a workaround can you read the csv from a python cell or read the csv with pandas and convert?

…

On Wed, 5 Aug 2020 at 20:01, Anton Kulaga ***@***.***> wrote: A simple code like: val df = spark.read.format("csv").option("header", true).option("sep", "\t").load("/data/ensembl/99/ensembl_anage_vertebrates.tsv") df.printSchema() runs for ages in polynote for spark 3 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#926 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJSZO2O4L5WXTNQUNOSDUC3R7GT7ZANCNFSM4OCTXH6A> .

[antonkulaga]

I managed to make csv reading work with spark 3. Not sure how. If others do not have issues with spark 3 I am ok to close the issue

[kumarvc]

@antonkulaga
I am also facing similar issue with spark 3.

polynote.kernel.remote.SocketTransport$ProcessDied$: Kernel died unexpectedly

Uncaught exception: Kernel died unexpectedly (polynote.kernel.remote.SocketTransport$ProcessDied$)

How it was resolved? It would be good to understand how it was resolved.

[antonkulaga]

As I am not having this issue now I will probably close