Skip to content

lampepfl/compiler-benchmark

 
 

Repository files navigation

JMH benchmarks for the Scala Compiler

Goal: define a set of JMH benchmarks for the compiler to help drive performance tuning and catch performance regressions.

Based on:

Structure

Project Benchmark of
compilation The equivalent of scalac ...
micro Finer grained parts of the compiler
jvm Pure Java benchmarks to demonstrate JVM quirks
infrastructure Code to persist benchmark results and metadata

Recipes

Learning about JMH options

sbt> compilation/jmh:run -help

Benchmarking compiler performance

  • (optional) add new Scala sources (say aardvark) to a new directory in src/main/corpus
compilation/jmh:run (Cold|Warm|Hot)CompilationBenchmark 
   -p source=(<subdir of corpus>|/path/to/src/dir|@/path/to/argsfile)
   -p extraArgs=-nowarn

Using aliases

Avoid the tedium of typing all that out with:

sbt> hot -psource=scalap 
sbt> cold -psource=better-files 

Changing Scala Version

sbt> set scalaVersion in ThisBuild := "2.12.0-ab61fed-SNAPSHOT"
sbt> set scalaHome in ThisBuild := Some(file("/code/scala/build/pack"))
sbt> set scalaHome in compilation := "2.11.1" // if micro project isn't compatible with "2.11.1"

Persisting results

  • Provide INFLUX_PASSWORD as an environment variable
  • Replace jmh/run with jmh:runMain scala.bench.UploadingRunner

Results will be uploading into an InfluxDB instance at https://scala-ci.typesafe.com/influx/

These results will be plotted in our Grafana dashboard

Note that the https://github.com/scala/compiler-benchq project, which will trigger benchmarks for merges and sets of commits that we're backtesting.

Collecting profiling data

sbt> .../jmh:run Benchmark -prof jmh.extras.JFR // Java Flight Recorder

Using Graal

Install Graal VM and JDK8 with JVMCI.

compilation/jmh:run CompileSourcesBenchmark 
    -jvm      /path/to/labsjdk1.8.0_92-jvmci-0.20/bin/java
    -jvmArgs -XX:+UnlockExperimentalVMOptions
    -jvmArgs -XX:+EnableJVMCI
    -jvmArgs -Djvmci.class.path.append=/path/to/graalvm-0.15-re/lib/jvmci/graal.jar
    -jvmArgs -Xbootclasspath/a:/path/to/graalvm-0.15-re/lib/truffle-api.jar
    -jvmArgs -Djvmci.Compiler=graal 
    -jvmArgs -XX:+UseJVMCICompiler 

Exporting an args file from SBT

$ curl https://gist.githubusercontent.com/retronym/78d016a3f10c62da2fd47cacac867f25/raw/65d9a1e8458d5984784ecf411d6c4d257bfdf0c1/ArgsFile.scala >  ~/.sbt/0.13/plugins/ArgsFile.scala
$ cd /code/someProject
$ sbt core/compile:argsFilePrint
[info] Set current project to root (in build file:/Users/jason/code/better-files/)
-deprecation
-encoding
UTF-8
-feature
-language:implicitConversions
-language:reflectiveCalls
-unchecked
-Xfatal-warnings
-Xlint
-Yinline-warnings
-Yno-adapted-args
-Ywarn-dead-code
-Xfuture
/Users/jason/code/better-files/core/src/main/scala/better/files/Cmds.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/File.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/Implicits.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/package.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/Scanner.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/ThreadBackedFileMonitor.scala
[success] Total time: 0 s, completed 02/09/2016 11:51:58 AM

Place this output into a file. The full path to that file can be passed to the -source= option to run the benchmark on that project: -psource=@/path/to/args/file

Releases

No releases published

Packages

No packages published

Languages

  • Scala 95.8%
  • Java 4.1%
  • Shell 0.1%