[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server #19047
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added the case ExecutorLostFailure which was previously not there, thus, the default case would be executed in which case, task would be marked as completed.
Apache Spark Pull Request - July 26, 2017
…es not create SparkContext Added a flag to check whether user has initialized Spark Context. If it is true, then we let Application Master unregister with Resource Manager else we do not.
… that does not create SparkContext" This reverts commit f454c89. "Merged another issue to this one by mistake"
Spark - July 28, 2017
[SPARK-21503]- Making Changes as per comments: Removed match case statement and replaced it with an if clause.
Spark - August 1, 2017
[SPARK-21503]: Reverting Unit Test Code - Not needed.
SPARK - August 24, 2017
… for launching daemons like History Server Adding new env variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode.
jerryshao
reviewed
Aug 25, 2017
| @@ -136,7 +136,8 @@ void addOptionString(List<String> cmd, String options) { | |||
|
|
|||
| Set<String> cp = new LinkedHashSet<>(); | |||
| addToClassPath(cp, appClassPath); | |||
|
|
|||
| addToClassPath(cp, getenv("SPARK_DAEMON_CLASSPATH")); | |||
There was a problem hiding this comment.
I think it would be better to only support this env variable for daemon process like HistoryServer, ExternalShuffleService and others, like what we did for SPARK_DAEMON_JAVA_OPTS. Currently with your fix normal Spark application will also honor this env variable.
There was a problem hiding this comment.
I agree with you on this. I have made the necessary changes.
… for launching daemons like History Server Reverted the previous code change and added the environment variable SPARK_DAEMON_CLASSPATH only for launching daemon processes.
pgandhi999
changed the title
pgandhi999
changed the title
|
ok to test |
|
Test build #81133 has finished for PR 19047 at commit
|
|
test this please |
|
LGTM. I'd like to see these daemons start using normal Spark configs like the applications do, but that's a separate, larger change... |
|
Test build #81141 has finished for PR 19047 at commit
|
|
+1 |
asfgit
pushed a commit
that referenced
this pull request
Aug 28, 2017
…for launching daemons like History Server History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server. Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode. ## How was this patch tested? Initially, history server start script would fail for the reason being that it could not find the required jars for launching the server in the java classpath. Same was true for running Master and Worker in standalone mode. By adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath, both the daemons(History Server, Standalone daemons) are starting up and running. Author: pgandhi <pgandhi@yahoo-inc.com> Author: pgandhi999 <parthkgandhi9@gmail.com> Closes #19047 from pgandhi999/master. (cherry picked from commit 24e6c18) Signed-off-by: Tom Graves <tgraves@yahoo-inc.com>
MatthewRBruce
pushed a commit
to Shopify/spark
that referenced
this pull request
Jul 31, 2018
…for launching daemons like History Server History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server. Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode. ## How was this patch tested? Initially, history server start script would fail for the reason being that it could not find the required jars for launching the server in the java classpath. Same was true for running Master and Worker in standalone mode. By adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath, both the daemons(History Server, Standalone daemons) are starting up and running. Author: pgandhi <pgandhi@yahoo-inc.com> Author: pgandhi999 <parthkgandhi9@gmail.com> Closes apache#19047 from pgandhi999/master. (cherry picked from commit 24e6c18) Signed-off-by: Tom Graves <tgraves@yahoo-inc.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server.
Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode.
How was this patch tested?
Initially, history server start script would fail for the reason being that it could not find the required jars for launching the server in the java classpath. Same was true for running Master and Worker in standalone mode. By adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath, both the daemons(History Server, Standalone daemons) are starting up and running.