UI: Netty version clash

[treo]

Issue Description

deeplearning4j-ui uses vert.x, which in turn uses netty.
Other modules, like datavec-hadoop or datavec-arrow also have transient dependencies on netty - often in very different versions.

This results in problems that show themselves as NoSuchMethodError or ClassNotFoundException.

The devious thing about it, is that it depends on the order of dependencies in the pom.xml file. If the incompatible version is brought in by something first, then the dl4j ui doesn't start and if deeplearning4j-ui is declared first, then libraries depending on the old version will have issues when they are used.

Version Information

• Deeplearning4j version: 1.0.0-beta6

Additional Information

To reproduce the bad behavior simply move the deeplearning4j-ui dependency in the examples (https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/pom.xml#L109-L114) after the hadoop-common dependency.

In the example case, it breaks even more havoc, as a few things suddenly aren't compile-able any more, as even more dependency version clashes are triggered.

Workaround

If anyone finds this while trying to get things to work. If you don't need the features of datavec-hadoop or datavec-arrow that depend on netty, then make sure to declare deeplearning4j-ui first.

If you have the same issue with any other library, you can find out what it is that is resulting in the version clash by running mvn dependency:tree and look what depends on netty. Then you can either exclude the transitive dependency, or you can use the <dependencyManagement> section to set a specific version (for more information about doing that see https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html).

Caveat: This only paints over the problem. If the other library can't work with the API of a different version, then you will run into problems too.

[agibsonccc]

@treo I think deeplearning4j-hadoop can be eliminated? I've found in konduit-serving which uses the same dependency tree, that all you have to do is do dependencyManagement on the netty versions. That might be enough? Either way, spark/play was a similar issue we had in the past.

[treo]

The problem with forcing a dependency here, is that just because this fixes the UI, it doesn't fix the underlying problem that two different versions of netty are required. In the Hadoop case it is 3.8, so forcing 4.x to be used instead is going to just push the problem to a different part of the application.

[saudet]

Another more permanent solution to this kind of issue could be to adopt OSGi...