[SPARK-2290] Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor

[YanTangZhai]

Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor

[AmplabJenkins]

Can one of the admins verify this patch?

[pwendell]

If we are going to remove this feature, we should just take the sparkHome field out of ApplicationDescription entirely.

[YanTangZhai]

The sarkHome field is taken out of ApplicationDescription entirely. Please review again. Thanks.

[pwendell]

LGTM pending tests, this is something that has confused people before, so I think it's best to just leave it out.

[pwendell]

Jenkins, test this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16401/

[pwendell]

Jenkins, retest this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16440/

[andrewor14]

Jenkins, test this please

[andrewor14]

Changes look reasonable to me. There were a few questions from the mailing list about this, so it'll be good to get this in.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

QA tests have started for PR 1244. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16518/consoleFull

[SparkQA]

QA results for PR 1244:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16518/consoleFull

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16518/

[andrewor14]

@YanTangZhai This doesn't compile... could you fix it?

[YanTangZhai]

I've fixed the compile problem. Please review and test again. Thanks very much.

[CodingCat]

instead of discarding sparkHome parameter entirely, shall we just prioritizing local SPARK_HOME env and pass the applicationDesc.sparkHome only if SPARK_HOME is not set in worker side?

[andrewor14]

@CodingCat We only want to do this if the driver shares the same directory structure as the executors. This is an assumption that is incorrect in many deployment settings. Really, we should have something like spark.executor.home that is not the same as SPARK_HOME.

I am not 100% sure if we can just rip this functionality out actually. I am under the impression that Mesos still depends on something like this, so we should double check before we remove it.

[CodingCat]

@andrewor14 yeah, I agree with you, I just thought in somewhere (document in the earlier versions? I cannot find it now), the user has to set this env variable? so I said prioritizing worker side SPARK_HOME, if this is not set, Spark will try to read application setup about SPARK_HOME (which may generates error if the directory structure is not the same)....

I also noticed this JIRA https://issues.apache.org/jira/browse/SPARK-2454 (left some comments there)