Reproducible builds, composing different CBT version and various impr…

…ovements

One large commit, because it is was hard to do these things in isolation or to separate them now.

CBT now knows how to load other versions of itself
 - Support for reproducible builds (!), by providing a CBT git URL and hash to tie build to
 - Support for composing builds using different CBT versions (!)
 - introduce (in compatibility/) Java interfaces all CBT versions need  to stay compatible with, so they can talk to each other. And put extension methods to these interfaces in cbt package object

Class loading
 - add some sanity checks for class loading
 - improve class loader invalidation to fix bugs
 - implement caching in Java land class loaders. In particular to prevent the system class loader to repeatedly generate ClassNotFound exceptions in each sink of the class loader DAG for non JDK classes (meaning major speed up for projects with many classes).
 - getting rid of transient class loader cache unifying into "persistent" one instead (which is still wrong as invalidation eventually needs to invalidate entire sub graphs of the class loading DAG, not single class loaders. Seems like we'll have to abandon the hashmap based approach and tie caching to dependency objects)

Other Caching
 - cache dependencies extracted from xml files, which was one major time killer, but invalidate cache when cbt changed (maven dependency user facing api needs simplification now!)
 - memorize last successful compile time in the file system rather than memory, to guard against unnecessary recompiling even across launches (or when using cbt direct)

Structural improvements
 - Factor out ClassLoaderCache on Java land into its own class.
 - Port MultiClassLoader to Java land, to better compose classloaders in NailgunLauncher.
 - Remove many global constants and variables (in object paths and in NailgunLauncher) and pass them through instead. Needed for composing of builds.
 - move more code from resolver into Lib for less entanglement with classes (needed to compatibility interfaces) and better re-usability
 - remove canBeCached. Everything can be cached now, but we need to be careful about correct invalidation.
 - remove build announcing produced jars. We can add if ever needed.
 - change callNullary to return exit code instead of Unit as preparation for next commit introducing "recursive"

ScalaTest
 - Makes ScalaTest support work (still a bit too inflexible, but mostly works well)

README.md

@@ -50,7 +50,7 @@ class Build(context: cbt.Context) extends PackageBuild(context){
   override def artifactId = "play-json-extensions"
   override def dependencies =
     super.dependencies :+
-    MavenRepository.central.resolve(
+    MavenResolver(context.cbtHasChanged,context.paths.mavenCache,MavenResolver.central).resolve(
       // encouraged way to declare dependencies
       ScalaDependency("com.typesafe.play", "play-json", "2.4.4"),
       MavenDependency("joda-time", "joda-time", "2.9.2")

build/build.scala

@@ -5,12 +5,14 @@ import scala.collection.immutable.Seq
 
 class Build(context: Context) extends BasicBuild(context){
   // FIXME: somehow consolidate this with cbt's own boot-strapping from source.
-  override def dependencies = super.dependencies :+ MavenRepository.central.resolve(
-    MavenDependency("net.incongru.watchservice","barbary-watchservice","1.0"),
-    MavenDependency("org.eclipse.jgit", "org.eclipse.jgit", "4.2.0.201601211800-r"),
-    MavenDependency("com.typesafe.zinc","zinc","0.3.9"),
-    ScalaDependency("org.scala-lang.modules","scala-xml","1.0.5")
-  )
+  override def dependencies = {
+    super.dependencies :+ MavenResolver(context.cbtHasChanged,context.paths.mavenCache,MavenResolver.central).resolve(
+      MavenDependency("net.incongru.watchservice","barbary-watchservice","1.0"),
+      MavenDependency("org.eclipse.jgit", "org.eclipse.jgit", "4.2.0.201601211800-r"),
+      MavenDependency("com.typesafe.zinc","zinc","0.3.9"),
+      ScalaDependency("org.scala-lang.modules","scala-xml","1.0.5")
+    ) :+ BinaryDependency(new File(System.getenv("CBT_HOME")+"/compatibility"), Seq())
+  }
   override def sources = Seq(
     "nailgun_launcher", "stage1", "stage2"
   ).map(d => projectDirectory ++ ("/" + d))

compatibility/ArtifactInfo.java

@@ -0,0 +1,7 @@
+package cbt;
+
+public interface ArtifactInfo extends Dependency{
+  public abstract String artifactId();
+  public abstract String groupId();
+  public abstract String version();
+}

compatibility/BuildInterface.java

@@ -0,0 +1,11 @@
+package cbt;
+import java.io.*;
+
+public interface BuildInterface extends Dependency{
+  public abstract BuildInterface copy(Context context);  // needed to configure builds
+  public abstract String show(); // needed for debugging
+  public abstract String scalaVersion(); // needed to propagate scalaVersion to dependent builds
+  public abstract String[] crossScalaVersionsArray(); // FIXME: this probably can't use Scala classes
+  public abstract BuildInterface finalBuild(); // needed to propagage through build builds. Maybe we can get rid of this.
+  public abstract File[] triggerLoopFilesArray(); // needed for watching files across composed builds
+}

compatibility/Context.java

@@ -0,0 +1,21 @@
+package cbt;
+import java.io.*;
+import java.util.concurrent.ConcurrentHashMap;
+
+// TODO: try to reduce the number of members
+public abstract class Context{
+  public abstract File projectDirectory();
+  public abstract File cwd();
+  public abstract String[] argsArray();
+  public abstract String[] enabledLoggersArray();
+  public abstract Long startCompat();
+  public abstract Boolean cbtHasChangedCompat();
+  public abstract String versionOrNull();
+  public abstract String scalaVersionOrNull(); // needed to propagate scalaVersion to dependendee builds
+  public abstract ConcurrentHashMap<String,Object> permanentKeys();
+  public abstract ConcurrentHashMap<Object,ClassLoader> permanentClassLoaders();
+  public abstract File cache();
+  public abstract File cbtHome();
+  public abstract File compatibilityTarget();
+  public abstract BuildInterface parentBuildOrNull();
+}

compatibility/Dependency.java

@@ -0,0 +1,10 @@
+package cbt;
+import java.io.*;
+
+public interface Dependency{
+  public abstract String show();
+  public abstract Boolean needsUpdateCompat();
+  public abstract Dependency[] dependenciesArray();
+  public abstract File[] dependencyClasspathArray();
+  public abstract File[] exportedClasspathArray();
+}

compatibility/Result.java

@@ -0,0 +1,11 @@
+/*
+package cbt;
+import java.io.*;
+public interface Result<T>{
+  public abstract Integer exitCode();
+  public abstract OutputStream out();
+  public abstract OutputStream err();
+  public abstract InputStream in();
+  public abstract T value();
+}
+*/

coursier/Coursier.scala

@@ -7,7 +7,7 @@ object Coursier{
       import coursier._
       val repositories = Seq(
         Cache.ivy2Local,
-        MavenRepository("https://repo1.maven.org/maven2")
+        MavenResolver("https://repo1.maven.org/maven2")
       )
 
       val start = Resolution(

nailgun_launcher/CBTUrlClassLoader.java

@@ -11,7 +11,7 @@ public String toString(){
       + "(\n  "
       + Arrays.toString(getURLs())
       + ",\n  "
-      + join("\n  ",getParent().toString().split("\n"))
+      + join("\n  ",(getParent() == null?"":getParent().toString()).split("\n"))
       + "\n)"
     );
   }
@@ -53,7 +53,7 @@ public CbtURLClassLoader(URL[] urls, ClassLoader parent){
     assertExist(urls);
   }
   public CbtURLClassLoader(URL[] urls){
-    super(urls);
+    super(urls, null);
     assertExist(urls);
   }
 }

nailgun_launcher/ClassLoaderCache2.java

@@ -0,0 +1,37 @@
+package cbt;
+
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import static java.io.File.pathSeparator;
+import static cbt.Stage0Lib.*;
+
+final class ClassLoaderCache2<T>{
+  ConcurrentHashMap<String,Object> keys;
+  ConcurrentHashMap<Object,T> values;
+
+  public ClassLoaderCache2(
+    ConcurrentHashMap<String,Object> keys,
+    ConcurrentHashMap<Object,T> values
+  ){
+    this.keys = keys;
+    this.values = values;
+  }
+
+  public T get( String key ){
+    return values.get(
+      keys.get( key )
+    );
+  }
+
+  public Boolean contains( String key ){
+    return keys.containsKey( key );
+  }
+
+  public T put( T value, String key ){
+    LockableKey2 keyObject = new LockableKey2();
+    keys.put( key, keyObject );
+    values.put( keyObject, value );
+    return value;
+  }
+}
+class LockableKey2{}