[FLINK-7840] [build] Shade netty in akka

[StephanEwen]

What is the purpose of the change

This change shade's Akka's dependency on Netty away.

Note: Akka itself cannot be shaded away, there is some magic in Scala and annotations that Scala adds to preserve some compile time information, which make shading impossible (results in inconsistent code). That creates some problems.

I tried several approaches to shading Akka's Netty:

1. Add all Akka dependencies to flink-runtime. The dependencies disappear due to shading, but the classes are present in the same namespace.
   This leads to problems. For example, tests pull in the regular akka dependencies as well (transitively from akka-testkit) and for some reason it creates problems with the mixing of classes loaded from flink-runtime and akka-actor. One could fix that by adding all other akka dependencies as provided wherever akka-testkit is used.
   Similarly, users that want to use akka have to add their own dependency to akka and will get a similar clash of classes.
   This can be summed up as: Adding a dependency to a shaded jar (hence removing the dependency) without relocating the classes creates problems with dependency management.

2. Add only akka-remote to the flink-runtime jar. That is the minimum we need to add to shade the Netty dependency. It softens the problem mentioned in (1), because only one of the akka dependencies is present in the flink-runtime jar, but it does not solve it completely.

We shade all akka dependencies into the flink-dist jar anyways, so anyone running a job with Flink will need to set all its akka dependencies to provided and assume Flink's version anyways (reloading in a different classloader through child-first loading as a workaround aside).

So from the user's perspective, akka is always provided. Child-first class loading can save the day for some users (allowing them to have a different akka version in the user code class loader than Flink has in the system class loader), but we should not strictly rely on that - its a pretty delicate thing.

Brief change log

• Set all akka compile time dependencies outside flink-runtime to provided.
• Add akka-remote_* , io.netty and org.uncommons.math to the shaded flink-runtime jar
• Relocate org.jboss.netty and org.uncommons.math
• Add Netty as an explicit dependency to flink-testutils for the network proxy and shade it there as well.
• Adds a check for unshaded Netty to the Travis CI script

Verifying this change

• The tests for this change pass on Travis.
• One can simply test this by manually building the change and starting a simple standalone cluster and running one of the sample jobs.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no): t removes dependencies
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know): It affects akka which is used for communication and thus for failure detection.

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[zentol]

Could you explain more why we need to shade this here? are we hiding some transitive netty dependency?

[StephanEwen]

Its not a transitive dependency, but a direct one. Test-utils depends on netty directly for the network connection proxy. Because Netty clashes often and such clashes have made tests unstable in a very subtle way, I wanted to hide netty here again to avoid such future problems

[zentol]

ah ok. How come we aren't using flink-shaded-netty for that?

[StephanEwen]

Good question, maybe @pnowojski can answer that... My guess is he was more familiar with the Netty 3 API (org.jboss.netty) and our shaded netty is Netty 4 (io.netty).

[zentol]

Why exactly do we need this? AFAIK you can shade transitive dependencies without promoting them.

[StephanEwen]

This adds akka-remote's dependencies as direct dependencies to flink-runtime (like for example reactive-streams).
Without promoting transitive dependencies, we would need to add all transitive dependencies of akka-remote to the shaded jar, because akka-remote is removed as a dependency which also removes its transitive dependencies.

We could do that, but since we cannot relocate many of them (Scala) I though to keep the set of dependencies that are "present but undeclared" small and treat akka-remote's dependencies like regular dependencies.

[StephanEwen]

We could also add all transitive dependencies of akka-remote to the shaded jar. Its a similar consideration as "add all akka, or only akka-remote".

[zentol]

+1

[StephanEwen]

Thanks for the review, merging...

[StephanEwen]

Merging as soon as Travis gives a green light on the rebased branch...

[StephanEwen]

Merged in 8595dad

[luryson]

Is there any way to shade all akka dependencies ? In my project, scala has been shaded in another jar, which cause the failure of flink's initilization.