[SPARK-17387][PYSPARK] Allow passing of args to gateway to configure JVM

[BryanCutler]

What changes were proposed in this pull request?

When not using spark-submit, user configuration for the JVM is ignored when creating the SparkContext. This is because the JVM is started before the SparkConf object is initialized. This change allows user to specify --conf args on the command line that will be used when launching the java gateway to configure the JVM.

How was this patch tested?

Started a plain Python shell with this command and verified the JVM was configured with the correct amount of memory.

SPARK_HOME=$PWD PYTHONPATH=python:python/lib/py4j-0.10.3-src.zip python - --conf spark.driver.memory=4g

Note, the additional '-' is needed for the cmd line args to not be processed when invoking the interpreter.

[BryanCutler]

@vanzin and @zjffdu , this is my take on a simplified solution that will allow configuring the JVM from the command line. You still won't be able to use these confs in the shell when creating the SparkContext, but I believe this is also how PySpark and spark-shell work also.

[vanzin]

@BryanCutler sorry, this is not what I meant. Guess I was confused about your comment on the bug.

See my original code on the bug:

>>> from pyspark import SparkContext
>>> from pyspark import SparkConf
>>> conf = SparkConf().set("spark.driver.memory", "4g")
>>> sc = SparkContext(conf=conf)

That's what I want to work.

[BryanCutler]

Hmmm, that won't work in the PySpark shell either though, even if you restart the SparkContext. If that's what you want though, @zjffdu 's PR addresses that.

[vanzin]

that won't work in the PySpark shell either though, even if you restart the SparkContext

That's a separate issue which should also be fixed (stopping a python SparkContext does not stop the underlying JVM), and kinda depends on the fix for this one too.

[BryanCutler]

Ok, I'll close this off then

[SparkQA]

Test build #65125 has finished for PR 15019 at commit 100e442.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.