[SPARK-2849] Handle driver configs separately in client mode #1845

andrewor14 · 2014-08-08T02:09:48Z

In client deploy mode, the driver is launched from within SparkSubmit's JVM. This means by the time we parse Spark configs from spark-defaults.conf, it is already too late to control certain properties of the driver's JVM. We currently ignore these configs in client mode altogether.

spark.driver.memory
spark.driver.extraJavaOptions
spark.driver.extraClassPath
spark.driver.extraLibraryPath

This PR handles these properties before launching the driver JVM. It achieves this by spawning a separate JVM that runs a new class called SparkSubmitDriverBootstrapper, which spawns SparkSubmit as a sub-process with the appropriate classpath, library paths, java opts and memory.

…xtra

This commit also fixes a deadly typo.

Right now it's not super obvious how to specify multiple java options, especially ones with white spaces.

…xtra

The previous code used to ignore all closing quotes if the same token also has an escaped double quote. For example, in -Dkey="I am the \"man\"" the last token contains both escaped quotes and valid quotes. This used to be interpreted as a token that doesn't have a closing quote when it actually does. This is fixed in this commit.

This is so that the way this is parsed and the way Java parses its java opts is consistent.

We previously never dealt with this correctly, in that we evaluated all backslashes twice, once when passing spark.*.extraJavaOptions into SparkSubmit, and another time when calling Utils.splitCommandString. This means we need to pass the raw values of these configs directly to the JVM without evaluating the backslashes when launching SparkSubmit. The way we do this is through a few custom environment variables. As of this commit, the user should follow the format outlined in spark-defaults.conf.template for spark.*.extraJavaOptions, and the expected java options (with quotes, whitespaces and backslashes and everything) will be propagated to the driver or the executors correctly.

... to highlight our new-found ability to deal with special values.

…xtra

SparkQA · 2014-08-08T02:14:44Z

QA tests have started for PR 1845. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18169/consoleFull

SparkQA · 2014-08-08T03:05:02Z

QA results for PR 1845:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18169/consoleFull

SparkQA · 2014-08-20T18:40:40Z

QA tests have started for PR 1845 at commit 24dba60.

This patch merges cleanly.

pwendell · 2014-08-20T18:49:39Z

bin/spark-class

+  export JAVA_OPTS
+  export OUR_JAVA_MEM
+  export SPARK_CLASS=1
+  shift # Ignore main class and use our own


Just so I understand - is this only ever used when the main class is SparkSubmit? It might be good to say that in the comment. The way it is now makes it sound like we could substitute a different class from the one that was specified, but in fact, all this does is ignore the class here and we hard code it in the bootstrapper.

SparkQA · 2014-08-20T20:40:41Z

Tests timed out after a configured wait of 120m.

pwendell · 2014-08-20T20:44:00Z

Jenkins, test this please.

SparkQA · 2014-08-20T20:50:31Z

QA tests have started for PR 1845 at commit 24dba60.

This patch merges cleanly.

SparkQA · 2014-08-20T21:00:30Z

QA tests have started for PR 1845 at commit bed4bdf.

This patch merges cleanly.

SparkQA · 2014-08-20T21:44:42Z

QA tests have finished for PR 1845 at commit 24dba60.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-08-20T21:52:33Z

QA tests have finished for PR 1845 at commit bed4bdf.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- shift # Ignore main class (org.apache.spark.deploy.SparkSubmit) and use our own

pwendell · 2014-08-20T22:01:32Z

Okay LGTM as an immediate fix - want to pull this into an RC.

In client deploy mode, the driver is launched from within `SparkSubmit`'s JVM. This means by the time we parse Spark configs from `spark-defaults.conf`, it is already too late to control certain properties of the driver's JVM. We currently ignore these configs in client mode altogether. ``` spark.driver.memory spark.driver.extraJavaOptions spark.driver.extraClassPath spark.driver.extraLibraryPath ``` This PR handles these properties before launching the driver JVM. It achieves this by spawning a separate JVM that runs a new class called `SparkSubmitDriverBootstrapper`, which spawns `SparkSubmit` as a sub-process with the appropriate classpath, library paths, java opts and memory. Author: Andrew Or <andrewor14@gmail.com> Closes #1845 from andrewor14/handle-configs-bash and squashes the following commits: bed4bdf [Andrew Or] Change a few comments / messages (minor) 24dba60 [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 08fd788 [Andrew Or] Warn against external usages of SparkSubmitDriverBootstrapper ff34728 [Andrew Or] Minor comments 51aeb01 [Andrew Or] Filter out JVM memory in Scala rather than Bash (minor) 9a778f6 [Andrew Or] Fix PySpark: actually kill driver on termination d0f20db [Andrew Or] Don't pass empty library paths, classpath, java opts etc. a78cb26 [Andrew Or] Revert a few changes in utils.sh (minor) 9ba37e2 [Andrew Or] Don't barf when the properties file does not exist 8867a09 [Andrew Or] A few more naming things (minor) 19464ad [Andrew Or] SPARK_SUBMIT_JAVA_OPTS -> SPARK_SUBMIT_OPTS d6488f9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 1ea6bbe [Andrew Or] SparkClassLauncher -> SparkSubmitDriverBootstrapper a91ea19 [Andrew Or] Fix precedence of library paths, classpath, java opts and memory 158f813 [Andrew Or] Remove "client mode" boolean argument c84f5c8 [Andrew Or] Remove debug print statement (minor) b71f52b [Andrew Or] Revert a few more changes (minor) 7d94a8d [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 3a8235d [Andrew Or] Only parse the properties file if special configs exist c37e08d [Andrew Or] Revert a few more changes a396eda [Andrew Or] Nullify my own hard work to simplify bash 0effa1e [Andrew Or] Add code in Scala that handles special configs c886568 [Andrew Or] Fix lines too long + a few comments / style (minor) 7a4190a [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 7396be2 [Andrew Or] Explicitly comment that multi-line properties are not supported fa11ef8 [Andrew Or] Parse the properties file only if the special configs exist 371cac4 [Andrew Or] Add function prefix (minor) be99eb3 [Andrew Or] Fix tests to not include multi-line configs bd0d468 [Andrew Or] Simplify parsing config file by ignoring multi-line arguments 56ac247 [Andrew Or] Use eval and set to simplify splitting 8d4614c [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash aeb79c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 2732ac0 [Andrew Or] Integrate BASH tests into dev/run-tests + log error properly 8d26a5c [Andrew Or] Add tests for bash/utils.sh 4ae24c3 [Andrew Or] Fix bug: escape properly in quote_java_property b3c4cd5 [Andrew Or] Fix bug: count the number of quotes instead of detecting presence c2273fc [Andrew Or] Fix typo (minor) e793e5f [Andrew Or] Handle multi-line arguments 5d8f8c4 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra c7b9926 [Andrew Or] Minor changes to spark-defaults.conf.template a992ae2 [Andrew Or] Escape spark.*.extraJavaOptions correctly aabfc7e [Andrew Or] escape -> split (minor) 45a1eb9 [Andrew Or] Fix bug: escape escaped backslashes and quotes properly... 1cdc6b1 [Andrew Or] Fix bug: escape escaped double quotes properly c854859 [Andrew Or] Add small comment c13a2cb [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra 8e552b7 [Andrew Or] Include an example of spark.*.extraJavaOptions de765c9 [Andrew Or] Print spark-class command properly a4df3c4 [Andrew Or] Move parsing and escaping logic to utils.sh dec2343 [Andrew Or] Only export variables if they exist fa2136e [Andrew Or] Escape Java options + parse java properties files properly ef12f74 [Andrew Or] Minor formatting 4ec22a1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra e5cfb46 [Andrew Or] Collapse duplicate code + fix potential whitespace issues 4edcaa8 [Andrew Or] Redirect stdout to stderr for python 130f295 [Andrew Or] Handle spark.driver.memory too 98dd8e3 [Andrew Or] Add warning if properties file does not exist 8843562 [Andrew Or] Fix compilation issues... 75ee6b4 [Andrew Or] Remove accidentally added file 63ed2e9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra 0025474 [Andrew Or] Revert SparkSubmit handling of --driver-* options for only cluster mode a2ab1b0 [Andrew Or] Parse spark.driver.extra* in bash 250cb95 [Andrew Or] Do not ignore spark.driver.extra* for client mode (cherry picked from commit b3ec51b) Signed-off-by: Patrick Wendell <pwendell@gmail.com>

This is an effort to bring the Windows scripts up to speed after recent splashing changes in #1845. Author: Andrew Or <andrewor14@gmail.com> Closes #2129 from andrewor14/windows-config and squashes the following commits: 881a8f0 [Andrew Or] Add reference to Windows taskkill 92e6047 [Andrew Or] Update a few comments (minor) 22b1acd [Andrew Or] Fix style again (minor) afcffea [Andrew Or] Fix style (minor) 72004c2 [Andrew Or] Actually respect --driver-java-options 803218b [Andrew Or] Actually respect SPARK_*_CLASSPATH eeb34a0 [Andrew Or] Update outdated comment (minor) 35caecc [Andrew Or] In Windows, actually kill Java processes on exit f97daa2 [Andrew Or] Fix Windows spark shell stdin issue 83ebe60 [Andrew Or] Parse special driver configs in Windows (broken)

This is an effort to bring the Windows scripts up to speed after recent splashing changes in apache#1845. Author: Andrew Or <andrewor14@gmail.com> Closes apache#2129 from andrewor14/windows-config and squashes the following commits: 881a8f0 [Andrew Or] Add reference to Windows taskkill 92e6047 [Andrew Or] Update a few comments (minor) 22b1acd [Andrew Or] Fix style again (minor) afcffea [Andrew Or] Fix style (minor) 72004c2 [Andrew Or] Actually respect --driver-java-options 803218b [Andrew Or] Actually respect SPARK_*_CLASSPATH eeb34a0 [Andrew Or] Update outdated comment (minor) 35caecc [Andrew Or] In Windows, actually kill Java processes on exit f97daa2 [Andrew Or] Fix Windows spark shell stdin issue 83ebe60 [Andrew Or] Parse special driver configs in Windows (broken) Conflicts: bin/spark-class2.cmd

This is an effort to bring the Windows scripts up to speed after recent splashing changes in #1845. Author: Andrew Or <andrewor14@gmail.com> Closes #2156 from andrewor14/windows-config-branch-1.1 and squashes the following commits: 00b9dfe [Andrew Or] [SPARK-3167] Handle special driver configs in Windows

In client deploy mode, the driver is launched from within `SparkSubmit`'s JVM. This means by the time we parse Spark configs from `spark-defaults.conf`, it is already too late to control certain properties of the driver's JVM. We currently ignore these configs in client mode altogether. ``` spark.driver.memory spark.driver.extraJavaOptions spark.driver.extraClassPath spark.driver.extraLibraryPath ``` This PR handles these properties before launching the driver JVM. It achieves this by spawning a separate JVM that runs a new class called `SparkSubmitDriverBootstrapper`, which spawns `SparkSubmit` as a sub-process with the appropriate classpath, library paths, java opts and memory. Author: Andrew Or <andrewor14@gmail.com> Closes apache#1845 from andrewor14/handle-configs-bash and squashes the following commits: bed4bdf [Andrew Or] Change a few comments / messages (minor) 24dba60 [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 08fd788 [Andrew Or] Warn against external usages of SparkSubmitDriverBootstrapper ff34728 [Andrew Or] Minor comments 51aeb01 [Andrew Or] Filter out JVM memory in Scala rather than Bash (minor) 9a778f6 [Andrew Or] Fix PySpark: actually kill driver on termination d0f20db [Andrew Or] Don't pass empty library paths, classpath, java opts etc. a78cb26 [Andrew Or] Revert a few changes in utils.sh (minor) 9ba37e2 [Andrew Or] Don't barf when the properties file does not exist 8867a09 [Andrew Or] A few more naming things (minor) 19464ad [Andrew Or] SPARK_SUBMIT_JAVA_OPTS -> SPARK_SUBMIT_OPTS d6488f9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 1ea6bbe [Andrew Or] SparkClassLauncher -> SparkSubmitDriverBootstrapper a91ea19 [Andrew Or] Fix precedence of library paths, classpath, java opts and memory 158f813 [Andrew Or] Remove "client mode" boolean argument c84f5c8 [Andrew Or] Remove debug print statement (minor) b71f52b [Andrew Or] Revert a few more changes (minor) 7d94a8d [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 3a8235d [Andrew Or] Only parse the properties file if special configs exist c37e08d [Andrew Or] Revert a few more changes a396eda [Andrew Or] Nullify my own hard work to simplify bash 0effa1e [Andrew Or] Add code in Scala that handles special configs c886568 [Andrew Or] Fix lines too long + a few comments / style (minor) 7a4190a [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 7396be2 [Andrew Or] Explicitly comment that multi-line properties are not supported fa11ef8 [Andrew Or] Parse the properties file only if the special configs exist 371cac4 [Andrew Or] Add function prefix (minor) be99eb3 [Andrew Or] Fix tests to not include multi-line configs bd0d468 [Andrew Or] Simplify parsing config file by ignoring multi-line arguments 56ac247 [Andrew Or] Use eval and set to simplify splitting 8d4614c [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash aeb79c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into handle-configs-bash 2732ac0 [Andrew Or] Integrate BASH tests into dev/run-tests + log error properly 8d26a5c [Andrew Or] Add tests for bash/utils.sh 4ae24c3 [Andrew Or] Fix bug: escape properly in quote_java_property b3c4cd5 [Andrew Or] Fix bug: count the number of quotes instead of detecting presence c2273fc [Andrew Or] Fix typo (minor) e793e5f [Andrew Or] Handle multi-line arguments 5d8f8c4 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra c7b9926 [Andrew Or] Minor changes to spark-defaults.conf.template a992ae2 [Andrew Or] Escape spark.*.extraJavaOptions correctly aabfc7e [Andrew Or] escape -> split (minor) 45a1eb9 [Andrew Or] Fix bug: escape escaped backslashes and quotes properly... 1cdc6b1 [Andrew Or] Fix bug: escape escaped double quotes properly c854859 [Andrew Or] Add small comment c13a2cb [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra 8e552b7 [Andrew Or] Include an example of spark.*.extraJavaOptions de765c9 [Andrew Or] Print spark-class command properly a4df3c4 [Andrew Or] Move parsing and escaping logic to utils.sh dec2343 [Andrew Or] Only export variables if they exist fa2136e [Andrew Or] Escape Java options + parse java properties files properly ef12f74 [Andrew Or] Minor formatting 4ec22a1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra e5cfb46 [Andrew Or] Collapse duplicate code + fix potential whitespace issues 4edcaa8 [Andrew Or] Redirect stdout to stderr for python 130f295 [Andrew Or] Handle spark.driver.memory too 98dd8e3 [Andrew Or] Add warning if properties file does not exist 8843562 [Andrew Or] Fix compilation issues... 75ee6b4 [Andrew Or] Remove accidentally added file 63ed2e9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-driver-extra 0025474 [Andrew Or] Revert SparkSubmit handling of --driver-* options for only cluster mode a2ab1b0 [Andrew Or] Parse spark.driver.extra* in bash 250cb95 [Andrew Or] Do not ignore spark.driver.extra* for client mode

This is an effort to bring the Windows scripts up to speed after recent splashing changes in apache#1845. Author: Andrew Or <andrewor14@gmail.com> Closes apache#2129 from andrewor14/windows-config and squashes the following commits: 881a8f0 [Andrew Or] Add reference to Windows taskkill 92e6047 [Andrew Or] Update a few comments (minor) 22b1acd [Andrew Or] Fix style again (minor) afcffea [Andrew Or] Fix style (minor) 72004c2 [Andrew Or] Actually respect --driver-java-options 803218b [Andrew Or] Actually respect SPARK_*_CLASSPATH eeb34a0 [Andrew Or] Update outdated comment (minor) 35caecc [Andrew Or] In Windows, actually kill Java processes on exit f97daa2 [Andrew Or] Fix Windows spark shell stdin issue 83ebe60 [Andrew Or] Parse special driver configs in Windows (broken)

andrewor14 added 27 commits August 4, 2014 13:12

Do not ignore spark.driver.extra* for client mode

250cb95

Parse spark.driver.extra* in bash

a2ab1b0

Revert SparkSubmit handling of --driver-* options for only cluster mode

0025474

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

63ed2e9

…xtra

Remove accidentally added file

75ee6b4

Fix compilation issues...

8843562

Add warning if properties file does not exist

98dd8e3

Handle spark.driver.memory too

130f295

Redirect stdout to stderr for python

4edcaa8

Collapse duplicate code + fix potential whitespace issues

e5cfb46

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

4ec22a1

…xtra

Minor formatting

ef12f74

Escape Java options + parse java properties files properly

fa2136e

Only export variables if they exist

dec2343

Move parsing and escaping logic to utils.sh

a4df3c4

This commit also fixes a deadly typo.

Print spark-class command properly

de765c9

Include an example of spark.*.extraJavaOptions

8e552b7

Right now it's not super obvious how to specify multiple java options, especially ones with white spaces.

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

c13a2cb

…xtra

Add small comment

c854859

Fix bug: escape escaped backslashes and quotes properly...

45a1eb9

This is so that the way this is parsed and the way Java parses its java opts is consistent.

escape -> split (minor)

aabfc7e

Minor changes to spark-defaults.conf.template

c7b9926

... to highlight our new-found ability to deal with special values.

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

5d8f8c4

…xtra

Handle multi-line arguments

e793e5f

Fix typo (minor)

c2273fc

andrewor14 mentioned this pull request Aug 8, 2014

[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode #1770

Closed

pwendell reviewed Aug 20, 2014
View reviewed changes

Change a few comments / messages (minor)

bed4bdf

asfgit closed this in b3ec51b Aug 20, 2014

andrewor14 mentioned this pull request Aug 26, 2014

[SPARK-3167] Handle special driver configs in Windows #2129

Closed

andrewor14 mentioned this pull request Aug 27, 2014

[SPARK-3167] Handle special driver configs in Windows (Branch 1.1) #2156

Closed

andrewor14 deleted the handle-configs-bash branch August 27, 2014 18:15

viirya added a commit to viirya/spark-1 that referenced this pull request Oct 19, 2023

rdar://113781872: Enable BosonShuffleManager in Spark 3.4 (apache#1845)

714b4f1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2849] Handle driver configs separately in client mode #1845

[SPARK-2849] Handle driver configs separately in client mode #1845

andrewor14 commented Aug 8, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 20, 2014

pwendell Aug 20, 2014

SparkQA commented Aug 20, 2014

pwendell commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

pwendell commented Aug 20, 2014

[SPARK-2849] Handle driver configs separately in client mode #1845

[SPARK-2849] Handle driver configs separately in client mode #1845

Conversation

andrewor14 commented Aug 8, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 20, 2014

pwendell Aug 20, 2014

Choose a reason for hiding this comment

SparkQA commented Aug 20, 2014

pwendell commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

pwendell commented Aug 20, 2014