Skip to content

Commit

Permalink
[SPARK-20365][YARN] Remove local scheme when add path to ClassPath.
Browse files Browse the repository at this point in the history
In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars started with "local" scheme), we will get inaccurate classpath for AM and containers. This is because we don't remove "local" scheme when concatenating classpath. It is OK to run because classpath is separated with ":" and java treat "local" as a separate jar. But we could improve it to remove the scheme.

Updated `ClientSuite` to check "local" is not in the classpath.

cc jerryshao

Author: Li Yichao <lyc@zhihu.com>
Author: Li Yichao <liyichao.good@gmail.com>

Closes #18129 from liyichao/SPARK-20365.
  • Loading branch information
liyichao authored and Marcelo Vanzin committed Jun 1, 2017
1 parent f7cf209 commit 640afa4
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -1275,7 +1275,8 @@ private object Client extends Logging {
if (sparkConf.get(SPARK_ARCHIVE).isEmpty) {
sparkConf.get(SPARK_JARS).foreach { jars =>
jars.filter(isLocalUri).foreach { jar =>
addClasspathEntry(getClusterPath(sparkConf, jar), env)
val uri = new URI(jar)
addClasspathEntry(getClusterPath(sparkConf, uri.getPath()), env)
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ class ClientSuite extends SparkFunSuite with Matchers with BeforeAndAfterAll
cp should not contain (uri.getPath())
}
})
cp should not contain ("local")
cp should contain(PWD)
cp should contain (s"$PWD${Path.SEPARATOR}${LOCALIZED_CONF_DIR}")
cp should not contain (APP_JAR)
Expand Down

0 comments on commit 640afa4

Please sign in to comment.