SPARK-1528 - spark on yarn, add support for accessing remote HDFS #1159
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
| @@ -547,4 +535,36 @@ object ClientBase extends Logging { | |||
| null | |||
| } | |||
|
|
|||
| // get the list of namenodes the user may access | |||
|
Build triggered. |
|
Build started. |
|
thanks for the review. If you have better verbage to use in in the docs let me know. I need to rebase this to latest master too, which I'll do shortly. |
|
Build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16032/ |
|
Jenkins, test this please |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
| <td><code>spark.yarn.access.namenodes</code></td> | ||
| <td>(none)</td> | ||
| <td> | ||
| A list of secure HDFS namenodes your Spark application is going to access. For example, `spark.yarn.access.namenodes=hdfs://nn1.com:8032,hdfs://nn2.com:8032`. The Spark application must have acess to the namenodes listed and Kerberos must be properly configured to be able to access them (either in the same realm or in a trusted realm). Spark acquires security tokens for each of the namenodes so that the Spark application can access those remote HDFS clusters. |
There was a problem hiding this comment.
LGTM.
BTW, I know it's the current style of the file, but these long lines could use some breaks.
|
merged into master and branch-1.1 |
Add a config (spark.yarn.access.namenodes) to allow applications running on yarn to access other secure HDFS cluster. User just specifies the namenodes of the other clusters and we get Tokens for those and ship them with the spark application. Author: Thomas Graves <tgraves@apache.org> Closes #1159 from tgravescs/spark-1528 and squashes the following commits: ddbcd16 [Thomas Graves] review comments 0ac8501 [Thomas Graves] SPARK-1528 - add support for accessing remote HDFS (cherry picked from commit 2c0f705) Signed-off-by: Thomas Graves <tgraves@apache.org>
Add a config (spark.yarn.access.namenodes) to allow applications running on yarn to access other secure HDFS cluster. User just specifies the namenodes of the other clusters and we get Tokens for those and ship them with the spark application. Author: Thomas Graves <tgraves@apache.org> Closes apache#1159 from tgravescs/spark-1528 and squashes the following commits: ddbcd16 [Thomas Graves] review comments 0ac8501 [Thomas Graves] SPARK-1528 - add support for accessing remote HDFS
* [CARMEL-6394] Implement column lineage tools * More nodes * add expand & union, fix aggregation * Use distinct cases * Unify logical and physical plan * Support logicalplan * typo * typo
Add a config (spark.yarn.access.namenodes) to allow applications running on yarn to access other secure HDFS cluster. User just specifies the namenodes of the other clusters and we get Tokens for those and ship them with the spark application.