[SPARK-2454] Do not ship spark home to Workers #1734
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
QA tests have started for PR 1734. This patch merges cleanly. |
| @@ -81,7 +81,8 @@ private[spark] class Worker( | |||
| @volatile var registered = false | |||
| @volatile var connected = false | |||
| val workerId = generateWorkerId() | |||
| val sparkHome = new File(Option(System.getenv("SPARK_HOME")).getOrElse(".")) | |||
| val sparkHome = | |||
| new File(sys.props.get("spark.test.home").orElse(sys.env.get("SPARK_HOME")).getOrElse(".")) | |||
There was a problem hiding this comment.
is this needed to make the tests work?
There was a problem hiding this comment.
Yeah, otherwise all tests with local-cluster fail
|
QA results for PR 1734: |
|
QA tests have started for PR 1734. This patch merges cleanly. |
|
LGTM pending tests... thanks @andrewor14! |
|
QA results for PR 1734: |
|
Thanks _ merged this |
asfgit
pushed a commit
that referenced
this pull request
Aug 2, 2014
`spark.test.home` was introduced in #1734. This is fine for SBT but is failing maven tests. Either way it shouldn't throw an NPE. Author: Andrew Or <andrewor14@gmail.com> Closes #1739 from andrewor14/fix-spark-test-home and squashes the following commits: ce2624c [Andrew Or] Do not throw NPE if spark.test.home is not set
asfgit
pushed a commit
that referenced
this pull request
Aug 2, 2014
`spark.test.home` was introduced in #1734. This is fine for SBT but is failing maven tests. Either way it shouldn't throw an NPE. Author: Andrew Or <andrewor14@gmail.com> Closes #1739 from andrewor14/fix-spark-test-home and squashes the following commits: ce2624c [Andrew Or] Do not throw NPE if spark.test.home is not set
xiliu82
pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
When standalone Workers launch executors, they inherit the Spark home set by the driver. This means if the worker machines do not share the same directory structure as the driver node, the Workers will attempt to run scripts (e.g. bin/compute-classpath.sh) that do not exist locally and fail. This is a common scenario if the driver is launched from outside of the cluster. The solution is to simply not pass the driver's Spark home to the Workers. This PR further makes an attempt to avoid overloading the usages of `spark.home`, which is now only used for setting executor Spark home on Mesos and in python. This is based on top of apache#1392 and originally reported by YanTangZhai. Tested on standalone cluster. Author: Andrew Or <andrewor14@gmail.com> Closes apache#1734 from andrewor14/spark-home-reprise and squashes the following commits: f71f391 [Andrew Or] Revert changes in python 1c2532c [Andrew Or] Merge branch 'master' of github.com:apache/spark into spark-home-reprise 188fc5d [Andrew Or] Avoid using spark.home where possible 09272b7 [Andrew Or] Always use Worker's working directory as spark home
xiliu82
pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
`spark.test.home` was introduced in apache#1734. This is fine for SBT but is failing maven tests. Either way it shouldn't throw an NPE. Author: Andrew Or <andrewor14@gmail.com> Closes apache#1739 from andrewor14/fix-spark-test-home and squashes the following commits: ce2624c [Andrew Or] Do not throw NPE if spark.test.home is not set
sunchao
pushed a commit
to sunchao/spark
that referenced
this pull request
Jun 2, 2023
…1734) Co-authored-by: Russell Spitzer <russell.spitzer@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When standalone Workers launch executors, they inherit the Spark home set by the driver. This means if the worker machines do not share the same directory structure as the driver node, the Workers will attempt to run scripts (e.g. bin/compute-classpath.sh) that do not exist locally and fail. This is a common scenario if the driver is launched from outside of the cluster.
The solution is to simply not pass the driver's Spark home to the Workers. This PR further makes an attempt to avoid overloading the usages of
spark.home, which is now only used for setting executor Spark home on Mesos and in python.This is based on top of #1392 and originally reported by @YanTangZhai. Tested on standalone cluster.