Hadoop 3 Support

[jonshea]

I have created a fork of wip-4.0 with preliminary Hadoop 3 support, https://github.com/jonshea/cascading/tree/wip-4.5-hadoop-3 . I would love to be able to include this in the cascading main-line releases. What’s the best way forward here? Do you want to make a wip-4.5 branch that I make a pull request against?

[cwensel]

I'll start a wip-4.5 branch then look at a merge this week.

[cwensel]

@jonshea

I'd prefer to drop Hadoop 2 entirely in 4.5, but keep 4.5 at feature parity with the previous releases.

that is, 4.5 is simply a Hadoop 3 release of 4.0, and 4.6 is a Hadoop 3 release of 4.1 etc. until we drop Hadoop 2 entirely.

not exactly sure this is the right idea, but it was historically very challenging to keep a shared codebase between the hadoop 1 and hadoop 2 releases (that is, not have any duplicate source files, but still compile the common files against the appropriate libraries). also, tests already run 4 hours with one hadoop release.

thus my next point, your Hadoop 3 sub-projects look like copies of Hadoop 2 source tree, this unfortunately loses 10 years of commit history.

so i'm thinking simply rename the hadoop 2 sub-projects to hadoop 3 and change the dependencies and add any hadoop 3 changes.

this of course leaves open the Tez question, I think advancing that version also makes sense in 4.5, but I fear dramatic changes across the Tez api making that another challenge.

[cwensel]

looks like we resolved this in fea3a7b.