Make sure extension work as an applet

[nicolaspayette]

Current status: everything loads fine, but certain method calls create exceptions. Examples:

Exception in thread "JobThread" java.lang.NoSuchMethodError: scala.collection.mutable.Set$.canBuildFrom()Lscala/collection/generic/CanBuildFrom;
    at org.nlogo.extensions.nw.jgrapht.Generator.importToNetLogo(Generators.scala:31)
    at org.nlogo.extensions.nw.jgrapht.Generator.generate(Generators.scala:52)
    at org.nlogo.extensions.nw.jgrapht.Generator.ringGraphGenerator(Generators.scala:56)
    at org.nlogo.extensions.nw.jgrapht.Primitives$RingGeneratorPrim$.createTurtles(Primitives.scala:49)
    at org.nlogo.extensions.nw.jgrapht.Primitives$RingGeneratorPrim$.createTurtles(Primitives.scala:42)
    at org.nlogo.extensions.nw.NetworkExtensionUtil$turtleCreatingCommand$class.perform(NetworkExtensionUtil.scala:97)
    at org.nlogo.extensions.nw.jgrapht.Primitives$RingGeneratorPrim$.perform(Primitives.scala:42)
    at org.nlogo.prim._extern.perform(_extern.java:54)
    at org.nlogo.nvm.Context.stepConcurrent(Context.java:91)
    at org.nlogo.nvm.ConcurrentJob.step(ConcurrentJob.java:82)
    at org.nlogo.job.JobThread.org$nlogo$job$JobThread$$runPrimaryJobs(JobThread.scala:143)
    at org.nlogo.job.JobThread$$anonfun$run$1.apply$mcV$sp(JobThread.scala:78)
    at org.nlogo.job.JobThread$$anonfun$run$1.apply(JobThread.scala:76)
    at org.nlogo.job.JobThread$$anonfun$run$1.apply(JobThread.scala:76)
    at scala.util.control.Exception$Catch.apply(Exception.scala:88)
    at org.nlogo.util.Exceptions$.handling(Exceptions.scala:41)
    at org.nlogo.job.JobThread.run(JobThread.scala:75)

And:

Exception in thread "JobThread" java.lang.NoSuchMethodError: scala.collection.JavaConverters$.asJavaCollectionConverter(Lscala/collection/Iterable;)Lscala/collection/JavaConverters$AsJavaCollection;
    at org.nlogo.extensions.nw.jung.Graph$class.getVertices(Graphs.scala:39)
    at org.nlogo.extensions.nw.jung.UndirectedGraph.getVertices(Graphs.scala:104)
    at edu.uci.ics.jung.algorithms.importance.BetweennessCentrality.computeBetweenness(BetweennessCentrality.java:75)
    at edu.uci.ics.jung.algorithms.importance.BetweennessCentrality.step(BetweennessCentrality.java:174)
    at edu.uci.ics.jung.algorithms.util.IterativeProcess.evaluate(IterativeProcess.java:68)
    at org.nlogo.extensions.nw.jung.Ranker$class.$init$(Algorithms.scala:31)
    at org.nlogo.extensions.nw.jung.Algorithms$BetweennessCentrality$.<init>(Algorithms.scala:96)
    at org.nlogo.extensions.nw.jung.UndirectedGraph.BetweennessCentrality(Graphs.scala:104)
    at org.nlogo.extensions.nw.jung.Primitives$BetweennessCentralityPrim$.report(Primitives.scala:54)
    at org.nlogo.extensions.nw.jung.Primitives$BetweennessCentralityPrim$.report(Primitives.scala:50)
    at org.nlogo.prim._externreport.report(_externreport.java:50)
    at org.nlogo.prim._setturtlevariable.perform(_setturtlevariable.java:35)
    at org.nlogo.nvm.Context.runExclusive(Context.java:119)
    at org.nlogo.nvm.ExclusiveJob.run(ExclusiveJob.java:56)
    at org.nlogo.nvm.Context.runExclusiveJob(Context.java:161)
    at org.nlogo.prim._ask.perform_1(_ask.java:65)
    at org.nlogo.prim._ask.perform(_ask.java:36)
    at org.nlogo.nvm.Context.stepConcurrent(Context.java:91)
    at org.nlogo.nvm.ConcurrentJob.step(ConcurrentJob.java:82)
    at org.nlogo.job.JobThread.org$nlogo$job$JobThread$$runPrimaryJobs(JobThread.scala:143)
    at org.nlogo.job.JobThread$$anonfun$run$1.apply$mcV$sp(JobThread.scala:78)
    at org.nlogo.job.JobThread$$anonfun$run$1.apply(JobThread.scala:76)
    at org.nlogo.job.JobThread$$anonfun$run$1.apply(JobThread.scala:76)
    at scala.util.control.Exception$Catch.apply(Exception.scala:88)
    at org.nlogo.util.Exceptions$.handling(Exceptions.scala:41)
    at org.nlogo.job.JobThread.run(JobThread.scala:75)

Hypothesis: it is still ProGuard related.

I first thought it might have something to do with NetLogo/NetLogo#163, but the extensions does compile fine against NetLogoLite.jar.

[nicolaspayette]

I am making progress, but it seems that I am playing Whac-A-Mole with ProGuard, here. I have already added:

-keep class scala.collection.JavaConverters {
    *;
}

-keep class scala.collection.JavaConverters$AsJavaCollection {
    *;
}

...and that got rid of one of the errors above.

But now, simple stuff like flattening an option (converting an optional iterable to an iterable that can be empty) gives me:

 java.lang.NoSuchMethodError: scala.collection.generic.GenericTraversableTemplate.flatten(Lscala/Function1;)Lscala/collection/GenTraversable;

So I am wondering if I should:

1. keep adding things to NetLogoLite.jar until I think I've covered all code paths
2. add bigger chunks of the scala library: maybe the whole collection package should be in NetLogoLite.jar anyway?
3. try to refactor all my code to avoid stuff that breaks with ProGuard.

Some things to consider:

• All three options would require lots of testing, and I don't know if there is a way to make sure we have covered all possibilities.
• 2 seems safer than 1, but I am not sure what difference it would make to the size of NetLogoLite.jar.
• Regarding 3, x.getOrElse(Iterable()) is a bit uglier than x.flatten, but that is no big deal. Some other things, like not being able to directly convert a scala.collection.Iterable to a java.util.Collection, would require ugly workarounds or bigger refactorings.
• These three options are not mutually exclusive: we can add some stuff to NetLogoLite and work around things that are easy to avoid.
• 1 and 2 require a new NetLogoLite.jar for using the NW extension in applets. Until NetLogo 5.0.3, it could be distributed with the extension (with appropriate documentation).
• 1 and 2 (but 2 more than 1) would make life easier for future extension developers. I think we had not seen these problems before because most extensions were developed with Java, and maybe those that were developed in Scala are not used much with applets?
• if we go with 3, things that don't work should be documented in the extensions API wiki.

[nicolaspayette]

(should not have been closed - that was an accidental click)

[nicolaspayette]

Adding:

-keep class scala.collection.** { *; }

pushes NetLogoLite.jar to 6.3MB, which is probably not what we want. Also, there might have been something else going on, but it took ProGuard 15 minutes to do its job.

[nicolaspayette]

Each of these allowed me to neutralize a particular exception:

-keep class scala.collection.JavaConverters { *; }
-keep class scala.collection.JavaConverters$AsJavaCollection { *; }
-keep class scala.collection.generic.GenericTraversableTemplate { *; }
-keep class scala.collection.mutable.Map { *; }
-keep class scala.Option$WithFilter { *; }

but after getting a NoSuchMethodException for scala.collection.mutable.MapLike$getOrElseUpdate(), further adding:

-keep class scala.collection.mutable.MapLike { *; }

causes this upon loading the applet:

Exception in thread "AWT-EventQueue-9" java.lang.NoClassDefFoundError: Could not initialize class org.nlogo.api.Version$
    at org.nlogo.lite.AppletPanel.<init>(AppletPanel.scala:49)
    at org.nlogo.lite.Applet$$anon$1.<init>(Applet.scala:35)
    at org.nlogo.lite.Applet.init2(Applet.scala:35)
basic: Removed progress listener: sun.plugin.util.GrayBoxPainter$GrayBoxProgressListener@221a5d08

Maybe something else is wrong.

[nicolaspayette]

Restarting Firefox actually got rid of that last exception. Still whac-a-molin', though.

[nicolaspayette]

With this:

-keep class scala.collection.JavaConverters { *; }
-keep class scala.collection.JavaConverters$AsJavaCollection { *; }
-keep class scala.collection.generic.GenericTraversableTemplate { *; }
-keep class scala.collection.mutable.Map$ { *; }
-keep class scala.Option$WithFilter { *; }
-keep class scala.collection.mutable.MapLike { *; }
-keep class scala.collection.mutable.Set$ { *; }
-keep class scala.Option { *; }
-keep class scala.math.Ordering$Long$ { *; }
-keep class scala.collection.IterableLike { *; }

everything appears to be working. Can't be a 100% sure I've covered all code paths, though.

It occurred to me, however: maybe compiling against NetLogoLite without the Scala library on the class path would catch some errors. Will try that.

[nicolaspayette]

Eclipse is, unsurprisingly I suppose, not letting me compile without a Scala library on the class path. I wonder if there is a way to do it with sbt.

[nicolaspayette]

Idea for testing code paths in applet: port code in test.txt to an actual NetLogo model I can run in applet.

[nicolaspayette]

After porting the unit test code to a test applet, I further added:

-keep class scala.collection.TraversableLike { *; }
-keep class scala.package$ { *; }

Not 100% sure everything is covered, but I am leaving it at that for now.

[nicolaspayette]

All these additions take NetLogoLite.jar to 5.7 MB (+2.1 MB) and to the .pack.gz to 1.3 MB (+793 kB).

Not sure if I should push this to 5.0.x.

[nicolaspayette]

For now, at least, we are not going to go with the bigger jar for 5.0.x.

I've added the alternate jars to the nw extension distribution zip. This is documented in the readme. Still need to automate its inclusion in the zip (#63).

[nicolaspayette]

There is now a wiki page with extra info about this: https://github.com/NetLogo/NW-Extension/wiki/Producing-an-alternate-NetLogoLite.jar