[SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by MesosClusterD…

[pmackles]

…ispatcher

What changes were proposed in this pull request?

Allow JVM max heap size to be controlled for MesosClusterDispatcher via SPARK_DAEMON_MEMORY environment variable.

How was this patch tested?

Tested on local Mesos cluster

[ArtRand]

Hello @pmackles, thanks for this. It would be set to the value of SPARK_DRIVER_MEMORY by default correct? What to you think about introducing a new envvar (SPARK_DISPATCHER_MEMORY) so that it can be individually configured? Only motivated by the fact that the Dispatcher is a pretty unique beast compared to the other Spark daemons.

[pmackles]

Hi @ArtRand - Based on my testing and interpretation of the code, SPARK_DRIVER_MEMORY has no affect on MesosClusterDispatcher. Heap size always winds up being set to -Xmx1G (the default).

I went with SPARK_DAEMON_MEMORY because that's what is used for the Master/Worker in a standalone cluster and I think Dispatcher is pretty similar in nature to those daemons. That said, I don't have a strong opinion and SPARK_DISPATCHER_MEMORY works fine for my use case (docker container running on mesos) so I am happy to switch it in the name of getting this PR accepted.

[pmackles]

@ArtRand - WDYT? I was going to switch it to SPARK_DISPATCHER_MEMORY but then I noticed that the other env vars for MesosClusterDispatcher or also prefixed with SPARK_DAEMON_* so I thought it might be better to keep the names consistent.

[ArtRand]

@pmackles, Yeah that sounds good. The default would stay the same, correct? Thanks for this!

[pmackles]

@ArtRand - yeah default would remain the same which is 1g

[ArtRand]

LGTM

[pmackles]

@felixcheung - any chance of getting this tiny change merged and included in the upcoming 2.2.1 release?

[felixcheung]

Jenkins, test this please

[felixcheung]

@susanxhuynh or anyone from the mesos side would you please review?

[felixcheung]

@pmackles perhaps you could email this to dev@spark.apache.org to get some visibility to this and hopefully someone else on the mesos side can review?

[SparkQA]

Test build #83648 has finished for PR 19515 at commit 33a8e68.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

LGTM, merging to master / 2.2.