Compile incrementally within jars

[jvican]

This idea was provided by @stuhood in https://contributors.scala-lang.org/t/asking-for-your-feedback-on-sbt-scala-center-announcement/738/19. This is a top priority for me, so I'll work on it within the next months before the release candidate happens. I would not be surprised if it requires API changes.

Feature description in Stu's own words:

ie, compiling directly to output jars, and then supporting incremental compile that consumes the slightly modified jars. The JVM hates classpaths containing loose classes in directories (anecdotally, jarring all inputs improved cold-build performance by 4x in one case due to drastically fewer syscalls and less IO), and this definitely affects Zinc when lots of modules are in play.

There is room for experimenting but the potential speedup seems juicy, especially in big projects. Anything that increases compilation speed should be treated as a priority for Zinc 1.0.

[smarter]

Note that this won't work on 2.12 as long as scala/bug#10295 isn't fixed.

[eed3si9n]

See also https://github.com/typesafehub/sbt-tojar

[eed3si9n]

@szeiger reminded me that PKZIP as a format supports appending at the end of the file, which means that when compiling A.scala, B.scala, and C.scala, and then recompiling A.scala with some change, we can keep adding new A.scala-generated-*.class files at the end of the file, while updating the index.
He also suggested an idea of using custom classloader to filter out the A.scala-generated-*.class during the re-compilation of A.scala instead of physically removing them from the *.jar.

[mkeskells]

I would suggest using the Zip File System Provider available within java7 and above, as an initial experement
This provide a full read/write filesytem using NIO2 http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html

If this is not fast enough and not fixable to be, then I am happy to assist with the lower level APIs, and I have some experience with parallelizing the JAR& ZIP apis within JDK tools, and would be happy to help

[jvican]

I've heard several Zinc useres telling me this is already supported by Zinc. See scalacenter/bloop#33 for more context.