[SPARK-3071] Increase default driver memory #7132

ilganeli · 2015-06-30T19:26:53Z

I've updated default values in comments, documentation, and in the command line builder to be 1g based on comments in the JIRA. I've also updated most usages to point at a single variable defined in the Utils.scala and JavaUtils.java files. This wasn't possible in all cases (R, shell scripts etc.) but usage in most code is now pointing at the same place.

Please let me know if I've missed anything.

Will the spark-shell use the value within the command line builder during instantiation?

AmplabJenkins · 2015-06-30T19:27:12Z

Merged build triggered.

AmplabJenkins · 2015-06-30T19:27:22Z

Merged build started.

SparkQA · 2015-06-30T19:30:44Z

Test build #36178 has started for PR 7132 at commit 2698a3d.

AmplabJenkins · 2015-06-30T19:32:14Z

Merged build triggered.

AmplabJenkins · 2015-06-30T19:32:22Z

Merged build started.

AmplabJenkins · 2015-06-30T19:47:28Z

Merged build finished. Test FAILed.

mengxr · 2015-06-30T20:39:22Z

+1 and ping @pwendell

mengxr · 2015-06-30T20:40:05Z

core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala


 import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.json4s.jackson.JsonMethods._


Please check Spark code style guide about how to organize imports: https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide

Fixed import ordering.

AmplabJenkins · 2015-06-30T20:47:12Z

Merged build triggered.

AmplabJenkins · 2015-06-30T20:47:18Z

Merged build started.

SparkQA · 2015-06-30T20:48:29Z

Test build #36191 has started for PR 7132 at commit 09ad698.

andrewor14 · 2015-06-30T21:27:13Z

@ilganeli I found a few more places

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/ClientArguments.scala#L120

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L491

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala#L167

https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala#L49

https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala#L278

https://github.com/apache/spark/blob/master/R/pkg/R/sparkR.R#L112

SparkQA · 2015-06-30T21:38:08Z

Test build #36178 has finished for PR 7132 at commit 2698a3d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-30T21:38:44Z

Merged build finished. Test PASSed.

SparkQA · 2015-06-30T22:53:52Z

Test build #36191 has finished for PR 7132 at commit 09ad698.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-30T22:54:04Z

Merged build finished. Test FAILed.

…st all locations instead of setting manually in each

AmplabJenkins · 2015-07-01T11:58:10Z

Merged build triggered.

AmplabJenkins · 2015-07-01T12:20:06Z

Merged build started.

SparkQA · 2015-07-01T12:23:55Z

Test build #36254 has started for PR 7132 at commit 39732a1.

SparkQA · 2015-07-01T14:35:01Z

Test build #36254 has finished for PR 7132 at commit 39732a1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T14:35:14Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-07-01T14:53:11Z

Merged build triggered.

AmplabJenkins · 2015-07-01T14:53:21Z

Merged build started.

SparkQA · 2015-07-01T14:56:23Z

Test build #36268 has started for PR 7132 at commit e5db35d.

SparkQA · 2015-07-01T17:49:22Z

Test build #36286 has started for PR 7132 at commit 26cc177.

mengxr · 2015-07-01T17:54:03Z

@andrewor14 This is a major change. Where should we document it for the 1.5 release?

andrewor14 · 2015-07-01T18:26:17Z

@pwendell This is a major change. Where should we document it for the 1.5 release?

JoshRosen · 2015-07-01T18:36:52Z

Maybe we can create a release notes Google Doc or a page on the Apache CWiki for holding a list of things which need to go into the release notes?

Ordering

AmplabJenkins · 2015-07-01T18:43:11Z

Merged build triggered.

AmplabJenkins · 2015-07-01T18:43:18Z

Merged build started.

SparkQA · 2015-07-01T18:46:35Z

Test build #36291 has started for PR 7132 at commit fd67721.

SparkQA · 2015-07-01T19:06:03Z

Test build #36282 has finished for PR 7132 at commit 26cc177.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T19:06:17Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-07-01T19:33:14Z

Merged build triggered.

AmplabJenkins · 2015-07-01T19:33:24Z

Merged build started.

SparkQA · 2015-07-01T19:34:42Z

Test build #36297 has started for PR 7132 at commit 271610b.

SparkQA · 2015-07-01T19:45:12Z

Test build #36286 has finished for PR 7132 at commit 26cc177.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T19:45:25Z

Merged build finished. Test FAILed.

SparkQA · 2015-07-01T20:36:34Z

Test build #36291 has finished for PR 7132 at commit fd67721.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T20:36:46Z

Merged build finished. Test FAILed.

SparkQA · 2015-07-01T21:28:18Z

Test build #36297 has finished for PR 7132 at commit 271610b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T21:28:34Z

Merged build finished. Test FAILed.

andrewor14 · 2015-07-02T00:19:21Z

@ilganeli looks like a legitimate test failure

AmplabJenkins · 2015-07-02T02:08:11Z

Merged build triggered.

AmplabJenkins · 2015-07-02T02:08:16Z

Merged build started.

SparkQA · 2015-07-02T02:11:41Z

Test build #36325 has started for PR 7132 at commit 4074164.

SparkQA · 2015-07-02T04:26:29Z

Test build #36325 has finished for PR 7132 at commit 4074164.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-02T04:27:02Z

Merged build finished. Test PASSed.

andrewor14 · 2015-07-02T06:10:51Z

Merging into master! Thanks @ilganeli. We'll update the release notes separately.

Updated default value for driver memory

2698a3d

Missed one doc update

19b6f25

mengxr reviewed Jun 30, 2015
View reviewed changes

Update SubmitRestProtocolSuite.scala

09ad698

Fixed import ordering.

Ilya Ganelin added 2 commits July 1, 2015 07:55

Created default value for DRIVER MEM in Utils that's now used in almo…

a6f7deb

…st all locations instead of setting manually in each

merge fix

39732a1

Fixed test failure

e5db35d

Update JavaUtils.java

fd67721

Ordering

Ilya Ganelin added 2 commits July 1, 2015 15:30

Test fix

273b6e9

Merge branch 'SPARK-3071' of github.com:ilganeli/spark into SPARK-3071

271610b

String fix

4074164

asfgit closed this in 3697232 Jul 2, 2015


		import org.apache.spark.{SparkConf, SparkFunSuite}
		import org.json4s.jackson.JsonMethods._

[SPARK-3071] Increase default driver memory #7132

[SPARK-3071] Increase default driver memory #7132

Conversation

ilganeli commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

mengxr commented Jun 30, 2015

mengxr Jun 30, 2015

Choose a reason for hiding this comment

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

andrewor14 commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

SparkQA commented Jul 1, 2015

mengxr commented Jul 1, 2015

andrewor14 commented Jul 1, 2015

JoshRosen commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

andrewor14 commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

andrewor14 commented Jul 2, 2015