[SPARK-7281] [yarn] Add option to set AM's lib path in client mode.

[vanzin]

No description provided.

[sryza]

LGTM. Did you try this out on a cluster?

[vanzin]

yes I did.

Submit:

$ spark-submit --conf "spark.yarn.am.extraLibraryPath=/foo" --master yarn-client...

AM:

# ps axwww | grep Executor
17935 ?        Ss     0:00 /bin/bash -c LD_LIBRARY_PATH=/foo:...

[SparkQA]

Test build #31440 has finished for PR 5813 at commit 1cb6f42.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • "public class " + className + extendsText + " implements java.io.Serializable
• This patch does not change any dependencies.

[srowen]

LGTM

[tgravescs]

I'm not at all against this, just pointing out we had a discussion on this before: #3409

What is the exact use case here for this that has caused this change in thinking?

[vanzin]

Depending on your Hadoop config, if you don't have the Hadoop native libraries in the AM's lib path, you get this (sorry the first few lines of the stack trace are missing from our bug report):

at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:269)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:297)
at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:43)
...

[vanzin]

If curious, the JNI-based group mapper is org.apache.hadoop.security.JniBasedUnixGroupsMapping.

[srowen]

@tgravescs are you OK with that? Seems like you agree with the change.

[tgravescs]

yes looks good.