[SPARK-4953][Doc] Fix the description of building Spark with YARN #3787
Conversation
sarutak
changed the title
|
Test build #24769 has finished for PR 3787 at commit
|
| @@ -60,20 +60,29 @@ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package | |||
| mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package | |||
| {% endhighlight %} | |||
|
|
|||
| For Apache Hadoop 2.x, 0.23.x, Cloudera CDH, and other Hadoop versions with YARN, you can enable the "yarn" profile and optionally set the "yarn.version" property if it is different from "hadoop.version". As of Spark 1.3, Spark only supports YARN versions 2.2.0 and later. | |||
| For Apache Hadoop 2.2.0 and later and Cloudera CDH 5 with YARN, you can enable the "yarn" profile and optionally set the "yarn.version" property if it is different from "hadoop.version". As of Spark 1.3, Spark only supports YARN versions 2.2.0 and later. | |||
There was a problem hiding this comment.
This is not only applicable to CDH 5+, so I'd revert that addition. What was removed with yarn-alpha was not really "Hadoop 0.23" support, although it kind of lines up with that. Why not remove this whole qualifying "For Apache Hadoop ..." phrase altogether? Also, do you mean Spark 1.2? what are you referring to in 1.3 otherwise?
|
Test build #24771 has finished for PR 3787 at commit
|
| # Cloudera CDH 5.1.X | ||
| mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0-cdh5.1.X -DskipTests clean package | ||
|
|
||
| # Cloudera CDEH 5.2.X or 5.3.X |
There was a problem hiding this comment.
This has a typo in "CDEH" and are also not runnable. I don't see much value in elaborating this example 3 more times.
(As a related aside, I would like to see less, not more, vendor stuff in Spark anyway. Adding just this text unduly favors Cloudera a tiny bit; the alternative is to write a bunch of other vendor combos here, which is going to turn into at least a maintenance headache. I already disagree with maintaining vendor versioning info in the project POM.)
|
Test build #24783 has finished for PR 3787 at commit
|
At the section "Specifying the Hadoop Version" In building-spark.md, there is description about building with YARN with Hadoop 0.23.
Spark 1.3.0 will not support Hadoop 0.23 so we should fix the description.