Browse files

[SPARK-1089] fix the regression problem on ADD_JARS in 0.9

copied from JIRA, reported by @ash211

"Using the ADD_JARS environment variable with spark-shell used to add the jar to both the shell and the various workers. Now it only adds to the workers and importing a custom class in the shell is broken.
The workaround is to add custom jars to both ADD_JARS and SPARK_CLASSPATH.
We should fix ADD_JARS so it works properly again.
See various threads on the user list:
(another one that doesn't appear in the archives yet titled "ADD_JARS not working on 0.9")"

The reason of this bug is two-folds

in the current implementation of SparkILoop.scala, the settings.classpath is not set properly when the process() method is invoked

the weird behaviour of Scala 2.10, (I personally thought it is a bug)

if we simply set value of a PathSettings object (like settings.classpath), the isDefault is not set to true (this is a flag showing if the variable is modified), so it makes the PathResolver loads the default CLASSPATH environment variable value to calculated the path (see

what we have to do is to manually make this flag set, (

Author: CodingCat <>

Closes #13 from CodingCat/SPARK-1089 and squashes the following commits:

8af81e7 [CodingCat] impose non-null settings
9aa2125 [CodingCat] code cleaning
ce36676 [CodingCat] code cleaning
e045582 [CodingCat] fix the regression problem on ADD_JARS in 0.9
  • Loading branch information...
CodingCat authored and pwendell committed Feb 27, 2014
1 parent 6ccd6c5 commit 345df5f4a9c16a6a87440afa2b09082fc3d224bd
Showing with 7 additions and 2 deletions.
  1. +7 −2 repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
@@ -182,8 +182,13 @@ class SparkILoop(in0: Option[BufferedReader], protected val out: JPrintWriter,
/** Create a new interpreter. */
def createInterpreter() {
- if (addedClasspath != "")
- settings.classpath append addedClasspath
+ require(settings != null)
+ if (addedClasspath != "") settings.classpath.append(addedClasspath)
+ // work around for Scala bug
+ val totalClassPath = SparkILoop.getAddedJars.foldLeft(
+ settings.classpath.value)((l, r) => ClassPath.join(l, r))
+ this.settings.classpath.value = totalClassPath
intp = new SparkILoopInterpreter

0 comments on commit 345df5f

Please sign in to comment.