[SPARK-2290] Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor #1392
Conversation
|
Can one of the admins verify this patch? |
|
Jenkins, test this please |
|
QA tests have started for PR 1392. This patch merges cleanly. |
|
QA results for PR 1392: |
|
Hey @YanTangZhai, on second thought I think we should keep the config, but not set it by default like we do currently. The user may have multiple installations of Spark on the same Worker machine, and "spark.home" previously provided them a way to pick among these installations. We should keep this functionality, but make it optional as opposed to forcing it on them. Also, since we no longer need "spark.home" anymore, it would be good to remove all occurrences of it to remove confusion (exception for backwards compatibility). However, this is slightly tricky because Mesos handles this differently from other modes. So I suggest this: I will take over from here, because this change seems a little more involved than we originally imagined it to be. How does that sound? |
|
Hi @andrewor14 , that's ok. Thanks. |
|
@YanTangZhai can you merge this up to master and make sure it is passing tests? Thanks. I spoke with @andrewor14 about #1472 and we agreed that having this patch would be good. We are right on a deadline, so if you can't up-merge it in the next day I can just fix it up and then merge it. |
When standalone Workers launch executors, they inherit the Spark home set by the driver. This means if the worker machines do not share the same directory structure as the driver node, the Workers will attempt to run scripts (e.g. bin/compute-classpath.sh) that do not exist locally and fail. This is a common scenario if the driver is launched from outside of the cluster. The solution is to simply not pass the driver's Spark home to the Workers. This PR further makes an attempt to avoid overloading the usages of `spark.home`, which is now only used for setting executor Spark home on Mesos and in python. This is based on top of #1392 and originally reported by YanTangZhai. Tested on standalone cluster. Author: Andrew Or <andrewor14@gmail.com> Closes #1734 from andrewor14/spark-home-reprise and squashes the following commits: f71f391 [Andrew Or] Revert changes in python 1c2532c [Andrew Or] Merge branch 'master' of github.com:apache/spark into spark-home-reprise 188fc5d [Andrew Or] Avoid using spark.home where possible 09272b7 [Andrew Or] Always use Worker's working directory as spark home
When standalone Workers launch executors, they inherit the Spark home set by the driver. This means if the worker machines do not share the same directory structure as the driver node, the Workers will attempt to run scripts (e.g. bin/compute-classpath.sh) that do not exist locally and fail. This is a common scenario if the driver is launched from outside of the cluster. The solution is to simply not pass the driver's Spark home to the Workers. This PR further makes an attempt to avoid overloading the usages of `spark.home`, which is now only used for setting executor Spark home on Mesos and in python. This is based on top of apache#1392 and originally reported by YanTangZhai. Tested on standalone cluster. Author: Andrew Or <andrewor14@gmail.com> Closes apache#1734 from andrewor14/spark-home-reprise and squashes the following commits: f71f391 [Andrew Or] Revert changes in python 1c2532c [Andrew Or] Merge branch 'master' of github.com:apache/spark into spark-home-reprise 188fc5d [Andrew Or] Avoid using spark.home where possible 09272b7 [Andrew Or] Always use Worker's working directory as spark home
Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor