benchmark dotty compiler

[lrytz]

Wip: https://github.com/scala/compiler-benchmark/compare/master...lrytz:dotty?expand=1

[lrytz]

This is currently crashing the Scala compiler due to scala/scala3#2670.

> hot -p source=/Users/luc/scala/compiler-benchmark/sandbox/Hello.scala
[info] Updating {file:/Users/luc/scala/compiler-benchmark/}infrastructure...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] 	* com.google.guava:guava:20.0 -> 21.0
[warn] Run 'evicted' to see detailed eviction warnings
[info] Updating {file:/Users/luc/scala/compiler-benchmark/}compilation...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 9 Java sources to /Users/luc/scala/compiler-benchmark/infrastructure/target/classes...
[info] Compiling 1 Scala source and 1 Java source to /Users/luc/scala/compiler-benchmark/compilation/target/scala-2.11/classes...
[error] error while loading package, class file '/Users/luc/.ivy2/cache/ch.epfl.lamp/dotty-compiler_0.1/jars/dotty-compiler_0.1-0.1.2-RC1.jar(dotty/tools/package.class)' is broken
[error] (class java.lang.ArrayIndexOutOfBoundsException/1710)
[error] error while loading Main, class file '/Users/luc/.ivy2/cache/ch.epfl.lamp/dotty-compiler_0.1/jars/dotty-compiler_0.1-0.1.2-RC1.jar(dotty/tools/dotc/Main.class)' is broken
[error] (class java.lang.ArrayIndexOutOfBoundsException/2130)
[error] two errors found
[error] (compilation/compile:compileIncremental) Compilation failed
[error] Total time: 4 s, completed Jun 3, 2017 4:18:59 PM

cc @smarter

[smarter]

Hi @lrytz ! @olafurpg is already working on benchmarking infrastructure for dotty, you guys should collaborate, see https://github.com/dotty-staging/compiler-benchmark

[lrytz]

Nice! We should get this running on our machine so we have comparable results.

[olafurpg]

Cool! I'd love to collaborate. Here is a rough summary of changes I made to the repo,

• classload the compiler to overcome issues like the ArrayIndexOutOfBoundsException above, we do the same for dotc and scalac, see https://github.com/lampepfl/compiler-benchmark/blob/57b1fdf58237b54e1f71bf451d13318c18d8b2d8/compilation/src/main/scala/base/BaseBenchmark.scala#L45. This allows us interface directly with the bootstrapped compiler https://github.com/lampepfl/compiler-benchmark/blob/57b1fdf58237b54e1f71bf451d13318c18d8b2d8/dotc-runtime/src/main/scala/compilerbenchmark/Benchmark.scala#L26
  • I almost managed to compile the benchmark project with Dotty but hit on a MatchError in the backend.
• unit tests, https://github.com/lampepfl/compiler-benchmark/blob/57b1fdf58237b54e1f71bf451d13318c18d8b2d8/compilation/src/test/scala/base/BenchmarkTest.scala
• two new corpora: paiges (460 LOC) and squants (4.573 LOC)
• accommodate uploading git metadata from both Scala and Dotty repo in UploadingOutputFormat
• runBatch command that runs all benchmarks for latest nightly version of 2.12 and dotty.

If you are interested in getting any of those changes into scala/compiler-benchmark, I may be able to pull out the necessary bits into a review-friendly PR.

[smarter]

I almost managed to compile the benchmark project with Dotty but hit on a MatchError in the backend.

It'd be great if you could open a bug report for that.

[olafurpg]

@smarter Will do. I spent a while trying to minimize that error, but then I got distracted with Scaladays preparations! It was related to Java annotations and default parameters.

[lrytz]

@olafurpg nice work! I think we should get everything merged here so that we can run the benchmarks on dotc and publish to the same database. We run through jenkins (https://scala-ci.typesafe.com/view/scala-bench/job/compiler-benchmark/528/parameters/), we'll have to do a few adjustments there too.

[olafurpg]

I just opened #31, which adds support to compile the benchmarks with dotty.