jupyter-scala with Spark does not see imports

[mmatloka]

I am running jupyter notebook on the machine where is available configured spark distribution with spark-submit , spark-defaults etc. I have running Python and R kernels but I have issues with jupyter-scala. I try to run the following code inside the notebook;

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import com.datastax.spark.connector._

val conf = new SparkConf(true)
val sc = new SparkContext("spark://127.0.0.1:7077", "test", conf)

but it seems the jars are not visible by it

Main.scala:29: object datastax is not a member of package com
 ; import org.apache.spark.SparkConf ; import org.apache.spark.SparkContext ; import com.datastax.spark.connector._ ; val conf = { () =>
                                                                                         ^
Main.scala:29: object apache is not a member of package org
 ; import org.apache.spark.SparkConf ; import org.apache.spark.SparkContext ; import com.datastax.spark.connector._ ; val conf = { () =>
              ^
Main.scala:30: not found: type SparkConf
new SparkConf(true) 
    ^
Main.scala:29: object apache is not a member of package org
 ; import org.apache.spark.SparkConf ; import org.apache.spark.SparkContext ; import com.datastax.spark.connector._ ; val conf = { () =>
                                                  ^
Main.scala:33: not found: type SparkContext
new SparkContext("spark://127.0.0.1:7077", "test", conf) 
    ^

I have tried to add before it

classpath.addPath("/var/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar")
classpath.add("datastax" % "spark-cassandra-connector" % "1.6.0-M1-s_2.11")

but nothing changes.

I am running jupyter-scala from https://git.io/vzhRi .

[drizham]

This is how I went about solving the issue I had (similar to you)
Installed Jupyter Toree from here: https://github.com/apache/incubator-toree

[mmatloka]

@drizham thx, but unfortunatelly toree supports only scala 2.10?

[alexarchambault]

@mmatloka @drizham Sorry, I've been kind of lacking time recently for jupyter-scala (although I and colleagues are using it quite a lot - with Spark in particular - at my $$ job).

This notebook illustrates how I usually use it to run things with Spark on a YARN cluster. I intend to make the doc better after the 1.0 final release of coursier, that jupyter-scala uses extensively.

[alexarchambault]

Also, you should follow the updated README to install it.

[dimon777]

I run into the same issue with Spark 2.3 and Scala 2.11. I used this repo to integrate with jupyter but seeing same error. Is there Toree agnostic way to integrate Spark+Scala and Jupyter?
Also spark assembly is no longer presented in spark distribution. Perhaps your README and This notebook has to be updated