[FLINK-5998] Un-fat Hadoop from Flink fat jar

[haohui]

This PR implements FLINK-5998.

It marks all Hadoop dependency in the dist jar as provided so that users can plug in the jars from their own Hadoop distribution.

[rmetzger]

I have tested this change locally, but it does not exactly what my intention was.

What I want is that the lib/ folder contains a flink-dist.jar with all the Flink runtime dependencies, and a flink-dist-hadoop.jar with the packaged Hadoop dependencies.
This way, users have a good out of the box experience, but they can use their own Hadoop jars easily.

[greghogan]

@rmetzger, how would users know which Hadoop dependencies to include in flink-dist-hadoop.jar? Would they be copying multiple component jars into lib/?

[rmetzger]

Yes, the idea is that users in some environments can even delete flink-dist-hadoop.jar entirely, and just configure the classpath correctly to point to the Hadoop lib folder of their hadoop distribution.

[greghogan]

@rmetzger thanks for the clarification. Sounds good!

[haohui]

@rmetzger -- thanks for the clarification. Do you think that it is sufficient to mark the dependency of flink-shaded-hadoop2 as provided and produces a dedicated jar for it?

[rmetzger]

Mh, not sure.
What I would try is updating the assembly descriptor(s) in the flink-dist project, so that it creates two assemblies.
Afaik you need to define all dependencies as regular deps in flink-dist, and then include subsets of them into different assemblies.

[haohui]

@rmetzger ping...would you mind taking a look?

[StephanEwen]

@haohui I think this goes into the right direction, but the flink-shaded-hadoop2-1.3.jar should be in /lib, not in opt. That way, we preserve the original behavior (Hadoop dependencies are available by default), but make it easy to swap/remove the Hadoop dependency.

[rmetzger]

Sorry for the delay. I'm now checking out the change....

[rmetzger]

Success!

I got it to work on my CDH 5.4.2 virtual machine with the following environment variables set

export HADOOP_CLASSPATH=/usr/lib/hadoop-yarn/*:/usr/lib/hadoop/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop/lib/*:/etc/hadoop/conf:/usr/lib/hadoop-hdfs/*

and it was using CDH's Hadoop version:

2017-05-04 10:49:58,243 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Hadoop version: 2.6.0-cdh5.4.2

I'll merge this change to master.