add spark.driver.memory to config docs

[nartz]

It took me a minute to track this down, so I thought it could be useful to have it in the docs.

I'm unsure if 512mb is the default for spark.driver.memory? Also - there could be a better value for the 'description' to differentiate it from spark.executor.memory.

[SparkQA]

Can one of the admins verify this patch?

[andrewor14]

Can you call this SparkContext to be consistent with other places in the docs?

[andrewor14]

LGTM. I'm surprised this isn't documented.

[ash211]

One note about this setting is that I'm not sure it works in all settings -- if you pass it to a driver as a parameter then it's too late to take effect (the JVM has already started).

./bin/spark-shell --driver-java-options "-Dspark.driver.memory=1576m" doesn't actually change driver memory.

Maybe the docs should mention where it does and doesn't work?

[nartz]

using ./bin/spark-submit --driver-memory 3g myscript.py on the command line works for me

[andrewor14]

@ash211 I believe it'll work actually, because we treat --driver-* arguments specially in bash before we launch the driver JVM.

[ash211]

@andrewor14 the --driver-memory parameter works but --driver-java-options "-Dspark.driver.memory=3g" doesn't. Do you expect both to work?

[andrewor14]

Ah sorry you're right, it will work only if it's --driver-java-options "-Xms3g -Xmx3g", because those will be passed directly to the java command when launching the JVM. Though in general --driver-java-options (and likewise spark.driver.extraJavaOptions) should not take in Spark properties, so I'm not sure if that's worth a mention in the docs.

[andrewor14]

Hey @nartz, can you also add spark.driver.extraClassPath, spark.driver.extraLibraryPath and spark.driver.extraJavaOptions? These were originally intended for cluster mode, but very recently we added support for these in client mode as well. That's probably why they weren't documented.

[nartz]

i can, but it seems maybe the scope of this pull-request is too small, as it needs to encapsulate and explain a bunch of new parameters popping up, and nuances about when to use spark.driver.memory (as i see many people having problems on the mailing list) - maybe we should just turn it into an issue that can be more comprehensive?

[andrewor14]

Ok, fair enough. Though none of these configs are actually new; they have been around since 1.0 but just never documented. I can merge this and file a separate JIRA for these other configs. Thanks @nartz.

[andrewor14]

Also could you add [Docs] to the title? It will help us organize our PRs.

[andrewor14]

Ok I merged this into master.