SPARK-1576 (Allow JAVA_OPTS to be passed as a command line parameter to YARN client)

[nishkamravi2]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[tgravescs]

see the comment on the jira. Also things should generally flow through the spark-submit script now.

[vanzin]

I think this is (now) already available as "--driver-java-options" (works both for yarn-client and yarn-cluster modes).

[nishkamravi2]

--driver-java-options would only work through the spark-submit script. If the developer invokes yarn.deploy.Client directly (as is common practice thusfar), they would simply pass java options (using -D) and the following code would allow spark options to be accepted:

for ((k, v) <- sys.props.filterKeys(_.startsWith("spark")))
{ JAVA_OPTS += "-D" + k + "=" + "\"" + v + "\"" }

This too was checked in by Patrick fairly recently (last Monday I think).

I would prefer creation of an explicit flag (--spark-java-options or --driver-java-options) for yarn.deploy.Client in order to be consistent with the rest of Spark UI, buthen Patrick has already checked in the code above and I guess he intends on keeping it.

[pwendell]

@nishkamravi2 hey, the older yarn client is being deprecated so we probably won't add new features to it. That is, if users want to take advantage of new options they will need to switch over to spark-submit. Since it's basically a drop-in replacement, the switch should be straightforward.

Is what you are trying to do not supported by spark-submit? If so, mind elaborating a bit more (because setting java options for the driver and executors are both supported)?

[vanzin]

--driver-java-options would only work through the spark-submit script.

Correct, and as Patrick mentions, it should be the way to submit spark jobs
going forward. I've been using it with Yarn client and cluster modes with
no issues.

Marcelo

[nishkamravi2]

Spark-submit looks neat and it does make sense to encourage use of a single primary interface for Spark invocation. Whether or not there is value in cleaning up the secondary interfaces may be subject to debate. However, if we plan on deprecating (and potentially retiring) the secondary interfaces, then I can see your point-of-view.

[tgravescs]

@nishkamravi2 have you tried the spark-submit script and can we close this?

[nishkamravi2]

@tgravescs Sorry, missed this one somehow. Yes, spark-submit is now fully tested. We could keep this one open for 0.92 potentially.

[tgravescs]

can we close this now? We have a 1.x branch now with spark-submit and will have a 1.1 release relatively soon. If someone really needs this in the 0.9.x line they can reopen.

[nishkamravi2]

Ok, closed.