[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode #1770

andrewor14 · 2014-08-04T20:23:43Z

We currently respect these configs only for cluster mode. From the names of the configs themselves, however, there is every reason for the user to believe that these should also apply for client mode. This currently does not work only because we're missing the code for it, and this PR adds this missing code.

vanzin · 2014-08-04T20:26:00Z

Hmmm... what does "respect" mean here though? In client mode, the vm running SparkSubmit is the same one running the driver, so it's too late to do anything with spark.driver.extraJavaOptions, for example.

SparkQA · 2014-08-04T20:29:28Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17870/consoleFull

andrewor14 · 2014-08-04T20:47:20Z

Good point... well, for spark.driver.extraJavaOptions we can manually add it to the system properties dynamically, and maybe we can do something similar for the other driver options too. However it looks like there's already existing logic for handling this in bin/spark-submit itself, though it doesn't seem to be working correctly. Investigating.

andrewor14 · 2014-08-04T21:08:37Z

Oops, looks like bin/spark-submit already handles this correctly.

However, the open issue is that if the user specifies spark.driver.* configs in spark-defaults.conf, they actually won't be processed in time. However, the user might expect these to be processed for all modes, so we still need to fix that. I have updated the JIRA and the title accordingly.

SparkQA · 2014-08-04T21:32:47Z

QA results for PR 1770:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17870/consoleFull

…xtra

SparkQA · 2014-08-06T04:39:19Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17990/consoleFull

SparkQA · 2014-08-06T04:44:21Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17992/consoleFull

SparkQA · 2014-08-06T04:47:34Z

QA results for PR 1770:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17992/consoleFull

andrewor14 · 2014-08-06T05:17:58Z

test this please

SparkQA · 2014-08-06T05:18:24Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17996/consoleFull

SparkQA · 2014-08-06T05:29:56Z

QA results for PR 1770:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17990/consoleFull

SparkQA · 2014-08-06T06:04:00Z

QA results for PR 1770:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17996/consoleFull

andrewor14 · 2014-08-06T06:19:35Z

test this please

SparkQA · 2014-08-06T06:24:39Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18004/consoleFull

SparkQA · 2014-08-06T07:29:17Z

QA results for PR 1770:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18004/consoleFull

…xtra

andrewor14 · 2014-08-07T06:34:02Z

test this please

andrewor14 · 2014-08-07T06:37:17Z

Before my latest set of commits we were not parsing quoted arguments (with whitespaces, quotes and backslashes and everything) properly. This is now fixed as of the latest changes, and I have included an example of what we expect from spark.*.extraJavaOptions in spark-defaults.conf.

SparkQA · 2014-08-07T06:44:44Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18106/consoleFull

pwendell · 2014-08-07T06:45:38Z

One proposal - maybe we can just say in the docs that explain spark-defaults.conf that we don't support multi-line options. I think the java properties format is very liberal in what it accepts, we can probably (at least initially) be more conservative here.

SparkQA · 2014-08-07T07:37:12Z

QA results for PR 1770:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18106/consoleFull

andrewor14 · 2014-08-07T07:38:27Z

Well, we technically do support multi-line option, except for the very specific case of spark.driver.extra* in client mode. Maybe it's still worth a comment?

By the way, please don't merge this yet. I still need to verify whether the existing way of parsing these extra java options in scala is not affected.

The previous code used to ignore all closing quotes if the same token also has an escaped double quote. For example, in -Dkey="I am the \"man\"" the last token contains both escaped quotes and valid quotes. This used to be interpreted as a token that doesn't have a closing quote when it actually does. This is fixed in this commit.

SparkQA · 2014-08-07T19:39:25Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18139/consoleFull

This is so that the way this is parsed and the way Java parses its java opts is consistent.

SparkQA · 2014-08-07T20:30:43Z

QA results for PR 1770:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18139/consoleFull

We previously never dealt with this correctly, in that we evaluated all backslashes twice, once when passing spark.*.extraJavaOptions into SparkSubmit, and another time when calling Utils.splitCommandString. This means we need to pass the raw values of these configs directly to the JVM without evaluating the backslashes when launching SparkSubmit. The way we do this is through a few custom environment variables. As of this commit, the user should follow the format outlined in spark-defaults.conf.template for spark.*.extraJavaOptions, and the expected java options (with quotes, whitespaces and backslashes and everything) will be propagated to the driver or the executors correctly.

... to highlight our new-found ability to deal with special values.

…xtra

SparkQA · 2014-08-08T00:19:46Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18160/consoleFull

SparkQA · 2014-08-08T01:10:32Z

QA results for PR 1770:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18160/consoleFull

andrewor14 · 2014-08-08T01:39:43Z

The size of this PR is growing beyond the original expectation. I will close this and open another one that describes the issue the new changes are solving more precisely.

SparkQA · 2014-08-08T01:44:43Z

QA tests have started for PR 1770. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18167/consoleFull

andrewor14 · 2014-08-08T02:12:10Z

Closing this in favor of #1845

SparkQA · 2014-08-08T02:35:35Z

QA results for PR 1770:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18167/consoleFull

- Change logs: https://github.pie.apple.com/pie/boson/compare/64ccd51b...66a7fea6 - Release notes: https://github.pie.apple.com/pie/boson/releases/tag/0.3.8

### What changes were proposed in this pull request? The pr aims to upgrade FasterXML jackson from 2.15.2 to 2.16.0. ### Why are the changes needed? New version that fix some bugs, release notes as follows: - 2.1.6.0 https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.16, eg: [Databind](https://github.com/FasterXML/jackson-databind) [#1770](FasterXML/jackson-databind#1770): Incorrect deserialization for BigDecimal numbers - 2.15.3 https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15.3, eg: [Databind](https://github.com/FasterXML/jackson-databind) [#3968](FasterXML/jackson-databind#3968): Records with additional constructors failed to deserialize The last upgrade occurred 6 months ago, #41414 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43859 from panbingkun/SPARK-45967. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

Do not ignore spark.driver.extra* for client mode

250cb95

andrewor14 changed the title ~~[SPARK-2849] Spark submit should respect --driver-* for client mode~~ [SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode Aug 4, 2014

andrewor14 changed the title ~~[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode~~ [WIP][SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode Aug 4, 2014

andrewor14 added 3 commits August 5, 2014 21:32

Parse spark.driver.extra* in bash

a2ab1b0

Revert SparkSubmit handling of --driver-* options for only cluster mode

0025474

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

63ed2e9

…xtra

andrewor14 changed the title ~~[WIP][SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode~~ [SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode Aug 6, 2014

andrewor14 added 3 commits August 5, 2014 21:36

Remove accidentally added file

75ee6b4

Fix compilation issues...

8843562

Add warning if properties file does not exist

98dd8e3

Handle spark.driver.memory too

130f295

Redirect stdout to stderr for python

4edcaa8

andrewor14 added 3 commits August 6, 2014 13:26

Collapse duplicate code + fix potential whitespace issues

e5cfb46

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

4ec22a1

…xtra

Minor formatting

ef12f74

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

c13a2cb

…xtra

Add small comment

c854859

Fix bug: escape escaped backslashes and quotes properly...

45a1eb9

This is so that the way this is parsed and the way Java parses its java opts is consistent.

andrewor14 added 4 commits August 7, 2014 14:24

escape -> split (minor)

aabfc7e

Minor changes to spark-defaults.conf.template

c7b9926

... to highlight our new-found ability to deal with special values.

Merge branch 'master' of github.com:apache/spark into submit-driver-e…

5d8f8c4

…xtra

Handle multi-line arguments

e793e5f

andrewor14 mentioned this pull request Aug 8, 2014

[SPARK-2849] Handle driver configs separately in client mode #1845

Closed

andrewor14 closed this Aug 8, 2014

andrewor14 deleted the submit-driver-extra branch August 8, 2014 02:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode #1770

[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode #1770

andrewor14 commented Aug 4, 2014

vanzin commented Aug 4, 2014

SparkQA commented Aug 4, 2014

andrewor14 commented Aug 4, 2014

andrewor14 commented Aug 4, 2014

SparkQA commented Aug 4, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

andrewor14 commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

andrewor14 commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

andrewor14 commented Aug 7, 2014

andrewor14 commented Aug 7, 2014

SparkQA commented Aug 7, 2014

pwendell commented Aug 7, 2014

SparkQA commented Aug 7, 2014

andrewor14 commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 8, 2014

andrewor14 commented Aug 8, 2014

SparkQA commented Aug 8, 2014

andrewor14 commented Aug 8, 2014

SparkQA commented Aug 8, 2014

[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode #1770

[SPARK-2849] bin/spark-submit should respect spark.driver.* for client mode #1770

Conversation

andrewor14 commented Aug 4, 2014

vanzin commented Aug 4, 2014

SparkQA commented Aug 4, 2014

andrewor14 commented Aug 4, 2014

andrewor14 commented Aug 4, 2014

SparkQA commented Aug 4, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

andrewor14 commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

andrewor14 commented Aug 6, 2014

SparkQA commented Aug 6, 2014

SparkQA commented Aug 6, 2014

andrewor14 commented Aug 7, 2014

andrewor14 commented Aug 7, 2014

SparkQA commented Aug 7, 2014

pwendell commented Aug 7, 2014

SparkQA commented Aug 7, 2014

andrewor14 commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 7, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 8, 2014

andrewor14 commented Aug 8, 2014

SparkQA commented Aug 8, 2014

andrewor14 commented Aug 8, 2014

SparkQA commented Aug 8, 2014