[SPARK-4651] Adding -Phadoop-2.4+ to compile Spark with newer versions of Hadoop

[oza]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This PR is messed up and needs to be rebased, at least, but see JIRA comments.

[oza]

@srowen Oh, my bad. I'm rebasing the patch.

[oza]

@srowen Updated. I ran Spark on YARN with Hadoop 2.5.2 by compiling this option.

[pwendell]

Hi there - this doesn't make sense because what if they release 2.6, 2.7, etc and it doesn't work with this profile?

[sryza]

This has come up before, and I thought we resolved that the hadoop-2.4 profile would be used for all Hadoop versions above 2.5, until a Hadoop version changes dependencies in such a way that we must add a new profile.

[pwendell]

Yes, I think that is what our docs say. So then this new profile isn't necessary

[oza]

Then, can I update the version of Hadoop used in -Phadoop-2.4 to 2.5.2 instead of adding new profile? It's latest stable version.

[pwendell]

In the docs we offically recommend "mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=VERSION -DskipTests clean package" for Hadoop 2.4.X and 2.5.X. I'd prefer this to modifying the name of this profile. It's just one more build flag.

[oza]

Make sense. Closing this ticket. Thanks for your comments, Patrick, Sandy, and Sean!

[pwendell]

No problem.