[FLINK-924] Add automatic dependency retrieval for JarFileCreator

[qmlmoon]

This improvement sets the package of user code as default package and let user to specify additional packages to be scanned.
Tested with inner class, anonymous class and additional package.
The current API could be used like:

JarFileCreator jfc = new JarFileCreator(jarFile);
jfc.addClass(X.class).addPackage("com.project").createJarFile();

[StephanEwen]

Wow, this looks like good stuff!

But: It has not a single test that covers the building, verifying that wanted dependencies ade fully added (includings things like inner classes, static references, all that) and that unwanted ones are not added.

I think we can only merge this once it has some tests.

[StephanEwen]

Cool, the tests are a good start! I think it would be very good to add the following cases, that are tricky for the dependency visitor:

• An anonymous inner class in a static method that accesses a local variable in its closure
• An anonymous inner class in a non-static method that accesses a local variable in its closure
• An anonymous inner class in a non-static method that accesses a field of the enclosing object
• An anonymous inner class in an anonymous inner class that that accesses a field of the outermost enclosing object

[StephanEwen]

Also: What is the impact of Java 8 lambdas on this. Would they work, i.e. would the dependency visitor find their dependencies?

[qmlmoon]

Tests looks interesting! Java 8 lambdas is not tested. I'll test it in the next update.

[qmlmoon]

Hey @StephanEwen , i'm a bit confused about your cases. Could you also post the correct dependencies here.

[StephanEwen]

@qmlmoon Here are a few more details of what I think would be good to change / add as tests:

( updated, I forgot the core point ;-) )

• The ASM dependency you added conflicts with other transitive ASM depencencies, which are 4.x . We need a dependency management entry for this, and verify that the other dependencies (that themselves depend on ASM 4.x) work well with that higher version of ASM.
• A test in the java8 project that uses a lambda with dependency:

public class ThresholdFilter {
  public static boolean filter(Long value) {
    return value > 10;
  }
}

...
FilerFunction<Long> filter = (v) -> ThresholdFilter.filter(v);
DataSet<Long> = someData.filter(filter);
...

as well as

...
DataSet<Long> = someData.filter( (v) -> ThresholdFilter.filter(v) );
...

and

...
class UtilFunctions {
  public static FilterFunction<Long> getThresholdFilter(long threshold) {
    return (v) -> ThresholdFilter.filter(v);
  }
}

public static void main(String[] args) {
  ...
  DataSet<Long> = someData.filter(UtilFunctions.getThresholdFilter(10));
  ...
}
...

• The last case could be added also as a test for functions through inner classes.

[qmlmoon]

I tested it with Java 8 lambda today. ASM 4.x doesn't support Java 8. ASM 5 works as expected. I also checked other module that contains ASM dependency. It turns out that only flink-scala has a dependency to ASM 4.0, but I can't find a place that this library is used. Does anyone know where is this library used?

[aljoscha]

ASM 4.x was used in the ClosureCleaner that was removed in the rewrite of the Scala API. I have an open pull request that re-adds the ClosureCleaner: #156. I will see whether this also works with ASM 5.x, then we can bump the version to this.

[aljoscha]

Ok, seems to also work with ASM 5, so you can change the version there.

[StephanEwen]

I think we need to remove the versions in both places and add a depedencyManagement entry in the root pom.xml file for this to work safely.

[aljoscha]

Hi @qmlmoon,
sorry for the long wait on this PR. Could you please rebase on top of the current master and also get rid of the merge commits in the process?

[qmlmoon]

I rebased the PR on the current master and removed the annoying merge commits.

[aljoscha]

Thanks for working with this on me. Very nice contribution. 😄

[StephanEwen]

Very good!

Is there a way we can integrate this deeper with the remote environment? Something like env.autoCollectDependencies() and then it internally build the jar by itself?