Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spark 3.x support #926

Closed
antonkulaga opened this issue Jun 19, 2020 · 12 comments
Closed

spark 3.x support #926

antonkulaga opened this issue Jun 19, 2020 · 12 comments
Labels
enhancement New feature or request kernel spark

Comments

@antonkulaga
Copy link

Spark 3.x is released ( https://spark.apache.org/releases/spark-release-3-0-0.html ), would be cool to have it in polynote

@fivejjs
Copy link

fivejjs commented Jun 20, 2020

This is a desirable feature.

@aaronzo
Copy link

aaronzo commented Jun 25, 2020

@antonkulaga What exactly doesn't work for you? I've just installed spark 3 and used the latest polynote-2.12.tar.gz release which uses scala 2.12, the same version that spark 3.0.0 uses, and All is running swimmingly so far.

My setup is just a local dev setup though, with spark running with --master local[*] on WSL in Windows. I have not tested it on any meaningfully sized spark clusters yet.

For the record, I'm also using GraalVM with Java 11, though I doubt that makes any difference

@hohonuuli
Copy link

Spark 3 in polynote 0.3.11 seems to work just fine. You can try it out in Docker:

docker run -p 8192:8192 -v $HOME/Documents/Notebooks:/opt/polynote/notebooks --name polynote mbari/polynote:0.3.11

Be sure to change the path of the volume $HOME/Documents/Notebooks to the path of a directory on your local computer.

@marrocksd
Copy link

image

The kernel cant start if I try to add some spark properties in UI

@antonkulaga
Copy link
Author

I am having issues like:

polynote.kernel.remote.SocketTransport$ProcessDied$: Kernel died unexpectedly

Uncaught exception: Kernel died unexpectedly (polynote.kernel.remote.SocketTransport$ProcessDied$)

@antonkulaga
Copy link
Author

I managed to sovle kernel-died error, but now reading csv-s do not work. They run for ages without providing the results

@antonkulaga
Copy link
Author

A simple code like:

val df = spark.read.format("csv").option("header", true).option("sep", "\t").load("/data/ensembl/99/ensembl_anage_vertebrates.tsv")
df.printSchema()

runs for ages in polynote for spark 3

@aaronzo
Copy link

aaronzo commented Aug 8, 2020 via email

@antonkulaga
Copy link
Author

I managed to make csv reading work with spark 3. Not sure how. If others do not have issues with spark 3 I am ok to close the issue

@kumarvc
Copy link

kumarvc commented Aug 15, 2020

@antonkulaga
I am also facing similar issue with spark 3.

polynote.kernel.remote.SocketTransport$ProcessDied$: Kernel died unexpectedly

Uncaught exception: Kernel died unexpectedly (polynote.kernel.remote.SocketTransport$ProcessDied$)

How it was resolved? It would be good to understand how it was resolved.

@antonkulaga
Copy link
Author

As I am not having this issue now I will probably close

@Mehdi-Bendriss
Copy link

Mehdi-Bendriss commented Feb 15, 2021

@kumarvc in my case it was due to the fact that I was running Spark 3.0.1 (on EMR 6.2.0) with the Scala 2.11 Polynote build.
It was fixed when switching to the scala 2.12 Polynote release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request kernel spark
Projects
None yet
Development

No branches or pull requests

8 participants