Fix for TOREE-430 #129
Fix for TOREE-430 #129
Conversation
| @@ -61,6 +61,7 @@ trait ScalaInterpreterSpecific { this: ScalaInterpreter => | |||
| ): SparkIMain = { | |||
| val s = new SparkIMain(settings, out) | |||
| s.initializeSynchronous() | |||
| System.setProperty("spark.repl.class.outputDir", s.getClassOutputDirectory.getAbsolutePath) | |||
There was a problem hiding this comment.
Could we pass this as conf while creating the Spark Session?
There was a problem hiding this comment.
I've tried to figure out how to do that without heavy code modifications but haven't succeeded.
It worth noting that currently in a Scala 2.11 environment Toree sets this property in a similar manner. And in Scala 2.11 we can pass it directly to SparkConf with ease because spark.repl.class.outputDir's required value becomes known very early (when org.apache.spark.repl.Main is initialised).
But in a Scala 2.10 environment, things are different. spark.repl.class.outputDir' is initialised only with org.apache.spark.repl.SparkIMain, that is, after creation of ScalaInterpreter in Toree. But in order to pass spark.repl.class.outputDir directly to SparkConf we have to initialiseSparkIMain earlier than Toree's Kernel.
This is applicable for both Spark 2 and Spark 1.6.
|
@Myllyenko Could you please rebase and update your pr so we can make sure we have a clean build before merging the update. |
|
I started to work on an improved version of this modification (with more accurate splitting between 2.10/2.11 specifics). Unfortunately I didn't have enough time to finish it yet, but I hope it will be ready soon. |
…erty to a path to an output directory of a REPL compiler in the Scala 2.10 environment. (cherry picked from commit 72261a6) # Conflicts: # scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
|
I've rebased this PR. |
|
This PR is no longer relevant as it is Scala 2.10-specific. |
To fix TOREE-430, REPL-generated classes must be made visible to a classloader.
The only working way I found is to set the
spark.repl.class.outputDirsystem property (and subsequently the corresponding Spark configuration property) to a path to an output directory of a REPL compiler. To achieve that, I had to moveSparkIMaininitialisation before creation ofSparkContext.