Add JMH benchmarks and improve performance of Zinc phases #225
Conversation
|
Do you have benchmark to compare before & after? Preferably using shapeless or akka. |
| } | ||
|
|
||
| private class DependencyProcessor(unit: CompilationUnit) { | ||
| private def firstClassOrModuleDef(tree: Tree): Option[Tree] = { |
There was a problem hiding this comment.
Please, note that this code has been moved from the end of the file to the place where it's used. I decided not to change this implementation because there's not a collectFirst or find primitive in the reflect tree API and this is probably the most efficient implementation. I don't think it's worth it to reimplement it in a more functional style that doesn't leverage return.
| * class/trait/object declared in the compilation unit. Otherwise, issue warning. | ||
| */ | ||
| def processTopLevelImportDependency(dep: Symbol): Unit = { | ||
| if (!orphanImportsReported) { |
There was a problem hiding this comment.
We only report the orphan error once.
| // The dependency comes from a class file | ||
| binaryDependency(pf.file, binaryClassName) | ||
| case _ => | ||
| // TODO: If this happens, scala internals have changed. Log error. |
There was a problem hiding this comment.
This error is very unlikely to happen, but if it does it should be more than just devWarning or warning. That's why it's a todo.
There was a problem hiding this comment.
Can we at least throw an exception for now?
There was a problem hiding this comment.
I see that it is not a new logic so we can live without exception or error.
|
@eed3si9n Will do that between today and tomorrow. |
|
By the way, I would be happy if @romanowski has a look at this diff too. |
|
I've just finished running this on shapeless, and it spares two seconds in the incremental compilation. Before: After: As it's a micro-optimization, I think the results are good. #206 had a bigger impact (my initial guess), and in my opinion the changes of this PR bring us two things:
|
|
Nice. Thanks for testing. |
|
I'll keep this open for another day so @romanowski can review. |
|
Hmm I am a little bit skeptical about this 2 seconds since this is almost 5% of time spend in full build command (that contains 8 s spent on compiling compiler interface and I don't know how many on resolving dependencies). IIRC pure compilation time was around 20 s. 2 sec out of that means that compilation time is reduced by 10%. After my changes (on my machine) zinc generates around 5% of of overhead so it would mean that this fix speeds up non-zinc phases. @jvican can you print times spent in zinc phases? I suggest do following routine
Moreover I suggest using this changes: romanowski@afd5c04 -time spent in Dependency and API phase will be printed. |
| responsibleOfImports match { | ||
| case Some(classOrModuleDef) => | ||
| val sym = classOrModuleDef.symbol | ||
| val firstClassSymbol = if (sym.isModule) sym.moduleClass else sym |
There was a problem hiding this comment.
I think this can should be computed only once
| // The dependency comes from a class file | ||
| binaryDependency(pf.file, binaryClassName) | ||
| case _ => | ||
| // TODO: If this happens, scala internals have changed. Log error. |
There was a problem hiding this comment.
Can we at least throw an exception for now?
| // The dependency comes from a class file | ||
| binaryDependency(pf.file, binaryClassName) | ||
| case _ => | ||
| // TODO: If this happens, scala internals have changed. Log error. |
There was a problem hiding this comment.
I see that it is not a new logic so we can live without exception or error.
No, the 2 seconds are out of the full build, not the compilation time. You can see that in the logs I attached with the benchmarking results. Benchmarking logsI've benchmarked this PR again and these are the results: Before: After: HardwareThe machine used for the "benchmark" is Linux tribox 4.9.6-1-ARCH #1 SMP PREEMPT Thu Jan 26 09:22:26 CET 2017 x86_64 GNU/Linux. It has Turbo Boost disabled on purpose to get more confident results (note that the previous benchmarking has Turbo Boost enabled). Analysis of the resultsThe logs were achieved with the same methodology that @romanowski proposed. The first time My guess is that the first time is run Zinc is cold, and the second one is already hot and has already been optimized by the JVM. Anyhow, I think that this produces a significant speedup in cold starts and minor or zero speedups while hot. However, I prefer not to confirm this hypothesis and focus on other tasks of Zinc, since there's still some work more to do in the Note: this "benchmark" uses millis to compute the time of the phases, so please don't take it very seriously, this is just an experiment that shows measurable improvement. |
|
Can't call 0.447s to 0.481s a "minor speedup while hot"..
…On Fri, 10 Feb 2017, 08:29 Jorge, ***@***.***> wrote:
IIRC pure compilation time was around 20 s. 2 sec out of that means that
compilation time is reduced by 10%.
No, the 2 seconds are out of the full build, not the compilation time. You
can see that in the logs I attached with the benchmarking results.
Benchmarking logs
I've benchmarked this PR again and these are the results:
*Before*:
jvican in ~/shapeless [9:14:24]
> $ sbt [±sbt-1.0-stripped ●]
[info] Loading project definition from /home/jvican/shapeless/project (out-3)
[info] Set current project to root (in build file:/home/jvican/shapeless/) (out-3)
> clean
[info] Total time: 0 s, completed Feb 10, 2017 9:14:39 AM (out-23)
> update
[info] Updating {file:/home/jvican/shapeless/}core... (out-60)
[info] Resolving jline#jline;2.12.1 ... (out-60)
[info] Done updating. (out-60)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-70)
[info] Resolving jline#jline;2.12.1 ... (out-70)
[info] Done updating. (out-70)
[info] Updating {file:/home/jvican/shapeless/}root... (out-72)
[info] Resolving jline#jline;2.12.1 ... (out-72)
[info] Done updating. (out-72)
[info] Total time: 2 s, completed Feb 10, 2017 9:14:41 AM (out-49)
> compile
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-108)
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4 ... (out-108)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-112)
[info] Resolving jline#jline;2.12.1 ... (out-108)
[info] Done updating. (out-108)
[info] Updating {file:/home/jvican/shapeless/}root... (out-115)
[info] Resolving jline#jline;2.12.1 ... (out-115)
[info] Done updating. (out-115)
[info] API phase took : 2.354 s (out-78)
[info] Dependency phase took : 1.96 s (out-78)
[info] Done compiling (in 44.897 s (out-112)
[info] Total time: 47 s, completed Feb 10, 2017 9:15:29 AM (out-75)
> clean
[info] Total time: 1 s, completed Feb 10, 2017 9:15:36 AM (out-129)
> compile
[info] Updating {file:/home/jvican/shapeless/}core... (out-178)
[info] Resolving jline#jline;2.12.1 ... (out-178)
[info] Done updating. (out-178)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-190)
[info] Resolving jline#jline;2.12.1 ... (out-190)
[info] Done updating. (out-190)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-192)
[info] Updating {file:/home/jvican/shapeless/}root... (out-196)
[info] Resolving jline#jline;2.12.1 ... (out-196)
[info] Done updating. (out-196)
[info] API phase took : 1.774 s (out-156)
[info] Dependency phase took : 0.447 s (out-156)
[info] Done compiling (in 26.497 s (out-192)
[info] Total time: 28 s, completed Feb 10, 2017 9:16:05 AM (out-155)
*After*:
jvican in ~/shapeless [8:45:07]
> $ sbt [±sbt-1.0-stripped ●]
[info] Loading project definition from /home/jvican/shapeless/project (out-3)
[info] Set current project to root (in build file:/home/jvican/shapeless/) (out-3)
> clean
[info] Total time: 1 s, completed Feb 10, 2017 8:45:24 AM (out-23)
> update
[info] Updating {file:/home/jvican/shapeless/}core... (out-60)
[info] Resolving jline#jline;2.12.1 ... (out-60)
[info] Done updating. (out-60)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-70)
[info] Resolving jline#jline;2.12.1 ... (out-70)
[info] Done updating. (out-70)
[info] Updating {file:/home/jvican/shapeless/}root... (out-72)
[info] Resolving jline#jline;2.12.1 ... (out-72)
[info] Done updating. (out-72)
[info] Total time: 2 s, completed Feb 10, 2017 8:45:27 AM (out-49)
> compile
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-108)
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4 ... (out-108)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-112)
[info] Resolving jline#jline;2.12.1 ... (out-108)
[info] Done updating. (out-108)
[info] Updating {file:/home/jvican/shapeless/}root... (out-116)
[info] Resolving jline#jline;2.12.1 ... (out-116)
[info] Done updating. (out-116)
[info] API phase took : 2.554 s (out-78)
[info] Dependency phase took : 0.572 s (out-78)
[info] Done compiling (in 44.033 s (out-112)
[info] Total time: 46 s, completed Feb 10, 2017 8:46:15 AM (out-75)
> clean
[info] Total time: 0 s, completed Feb 10, 2017 8:46:17 AM (out-129)
> compile
[info] Updating {file:/home/jvican/shapeless/}core... (out-178)
[info] Resolving jline#jline;2.12.1 ... (out-178)
[info] Done updating. (out-178)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-188)
[info] Resolving jline#jline;2.12.1 ... (out-188)
[info] Done updating. (out-188)
[info] Updating {file:/home/jvican/shapeless/}root... (out-196)
[info] Resolving core#shapeless_2.11;2.3.3-SNAPSHOT ... (out-196)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-192)
[info] Resolving jline#jline;2.12.1 ... (out-196)
[info] Done updating. (out-196)
[info] API phase took : 1.58 s (out-156)
[info] Dependency phase took : 0.481 s (out-156)
[info] Done compiling (in 25.316 s (out-192)
[info] Total time: 27 s, completed Feb 10, 2017 8:46:46 AM (out-155)
Hardware
The machine used for the "benchmark" is Linux tribox 4.9.6-1-ARCH #1
<#1> SMP PREEMPT Thu Jan 26 09:22:26 CET
2017 x86_64 GNU/Linux. It has *Turbo Boost disabled* on purpose to get
more confident results (note that the previous benchmarking has Turbo Boost
enabled).
Analysis of the results
The logs were achieved with the same methodology that @romanowski
<https://github.com/romanowski> proposed. The first time compile was run,
this PR decreased the running time of Dependency by 3/4. The second time
it's run, performance is on par (this is very weird considering that I'm
cleaning the directory before running compile again).
My guess is that the first time is run Zinc is cold, and the second one is
already hot and has already been optimized by the JVM. Anyhow, I think that
this produces a significant speedup in cold starts and *minor speedups*
while hot. However, I prefer not to confirm this hypothesis and focus on
other tasks of Zinc, since there's still some work more to do in the
ExtractAPI and Dependency algorithm that I'm planning to change in my
next PR.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#225 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAVCIlzIKM22ZtfLPy7XZ6IL2X89Qks8ks5rbCACgaJpZM4L7da_>
.
|
|
Also, is 1 data point before and 1 data point after sufficient benchmarking
proof?
…On Fri, 10 Feb 2017, 09:17 Dale Wijnand, ***@***.***> wrote:
Can't call 0.447s to 0.481s a "minor speedup while hot"..
On Fri, 10 Feb 2017, 08:29 Jorge, ***@***.***> wrote:
IIRC pure compilation time was around 20 s. 2 sec out of that means that
compilation time is reduced by 10%.
No, the 2 seconds are out of the full build, not the compilation time. You
can see that in the logs I attached with the benchmarking results.
Benchmarking logs
I've benchmarked this PR again and these are the results:
*Before*:
jvican in ~/shapeless [9:14:24]
> $ sbt [±sbt-1.0-stripped ●]
[info] Loading project definition from /home/jvican/shapeless/project (out-3)
[info] Set current project to root (in build file:/home/jvican/shapeless/) (out-3)
> clean
[info] Total time: 0 s, completed Feb 10, 2017 9:14:39 AM (out-23)
> update
[info] Updating {file:/home/jvican/shapeless/}core... (out-60)
[info] Resolving jline#jline;2.12.1 ... (out-60)
[info] Done updating. (out-60)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-70)
[info] Resolving jline#jline;2.12.1 ... (out-70)
[info] Done updating. (out-70)
[info] Updating {file:/home/jvican/shapeless/}root... (out-72)
[info] Resolving jline#jline;2.12.1 ... (out-72)
[info] Done updating. (out-72)
[info] Total time: 2 s, completed Feb 10, 2017 9:14:41 AM (out-49)
> compile
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-108)
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4 ... (out-108)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-112)
[info] Resolving jline#jline;2.12.1 ... (out-108)
[info] Done updating. (out-108)
[info] Updating {file:/home/jvican/shapeless/}root... (out-115)
[info] Resolving jline#jline;2.12.1 ... (out-115)
[info] Done updating. (out-115)
[info] API phase took : 2.354 s (out-78)
[info] Dependency phase took : 1.96 s (out-78)
[info] Done compiling (in 44.897 s (out-112)
[info] Total time: 47 s, completed Feb 10, 2017 9:15:29 AM (out-75)
> clean
[info] Total time: 1 s, completed Feb 10, 2017 9:15:36 AM (out-129)
> compile
[info] Updating {file:/home/jvican/shapeless/}core... (out-178)
[info] Resolving jline#jline;2.12.1 ... (out-178)
[info] Done updating. (out-178)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-190)
[info] Resolving jline#jline;2.12.1 ... (out-190)
[info] Done updating. (out-190)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-192)
[info] Updating {file:/home/jvican/shapeless/}root... (out-196)
[info] Resolving jline#jline;2.12.1 ... (out-196)
[info] Done updating. (out-196)
[info] API phase took : 1.774 s (out-156)
[info] Dependency phase took : 0.447 s (out-156)
[info] Done compiling (in 26.497 s (out-192)
[info] Total time: 28 s, completed Feb 10, 2017 9:16:05 AM (out-155)
*After*:
jvican in ~/shapeless [8:45:07]
> $ sbt [±sbt-1.0-stripped ●]
[info] Loading project definition from /home/jvican/shapeless/project (out-3)
[info] Set current project to root (in build file:/home/jvican/shapeless/) (out-3)
> clean
[info] Total time: 1 s, completed Feb 10, 2017 8:45:24 AM (out-23)
> update
[info] Updating {file:/home/jvican/shapeless/}core... (out-60)
[info] Resolving jline#jline;2.12.1 ... (out-60)
[info] Done updating. (out-60)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-70)
[info] Resolving jline#jline;2.12.1 ... (out-70)
[info] Done updating. (out-70)
[info] Updating {file:/home/jvican/shapeless/}root... (out-72)
[info] Resolving jline#jline;2.12.1 ... (out-72)
[info] Done updating. (out-72)
[info] Total time: 2 s, completed Feb 10, 2017 8:45:27 AM (out-49)
> compile
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-108)
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4 ... (out-108)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-112)
[info] Resolving jline#jline;2.12.1 ... (out-108)
[info] Done updating. (out-108)
[info] Updating {file:/home/jvican/shapeless/}root... (out-116)
[info] Resolving jline#jline;2.12.1 ... (out-116)
[info] Done updating. (out-116)
[info] API phase took : 2.554 s (out-78)
[info] Dependency phase took : 0.572 s (out-78)
[info] Done compiling (in 44.033 s (out-112)
[info] Total time: 46 s, completed Feb 10, 2017 8:46:15 AM (out-75)
> clean
[info] Total time: 0 s, completed Feb 10, 2017 8:46:17 AM (out-129)
> compile
[info] Updating {file:/home/jvican/shapeless/}core... (out-178)
[info] Resolving jline#jline;2.12.1 ... (out-178)
[info] Done updating. (out-178)
[info] Updating {file:/home/jvican/shapeless/}scratch... (out-188)
[info] Resolving jline#jline;2.12.1 ... (out-188)
[info] Done updating. (out-188)
[info] Updating {file:/home/jvican/shapeless/}root... (out-196)
[info] Resolving core#shapeless_2.11;2.3.3-SNAPSHOT ... (out-196)
[info] Compiling 79 Scala sources to /home/jvican/shapeless/core/target/scala-2.11/classes... (out-192)
[info] Resolving jline#jline;2.12.1 ... (out-196)
[info] Done updating. (out-196)
[info] API phase took : 1.58 s (out-156)
[info] Dependency phase took : 0.481 s (out-156)
[info] Done compiling (in 25.316 s (out-192)
[info] Total time: 27 s, completed Feb 10, 2017 8:46:46 AM (out-155)
Hardware
The machine used for the "benchmark" is Linux tribox 4.9.6-1-ARCH #1
<#1> SMP PREEMPT Thu Jan 26 09:22:26 CET
2017 x86_64 GNU/Linux. It has *Turbo Boost disabled* on purpose to get
more confident results (note that the previous benchmarking has Turbo Boost
enabled).
Analysis of the results
The logs were achieved with the same methodology that @romanowski
<https://github.com/romanowski> proposed. The first time compile was run,
this PR decreased the running time of Dependency by 3/4. The second time
it's run, performance is on par (this is very weird considering that I'm
cleaning the directory before running compile again).
My guess is that the first time is run Zinc is cold, and the second one is
already hot and has already been optimized by the JVM. Anyhow, I think that
this produces a significant speedup in cold starts and *minor speedups*
while hot. However, I prefer not to confirm this hypothesis and focus on
other tasks of Zinc, since there's still some work more to do in the
ExtractAPI and Dependency algorithm that I'm planning to change in my
next PR.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#225 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAVCIlzIKM22ZtfLPy7XZ6IL2X89Qks8ks5rbCACgaJpZM4L7da_>
.
|
|
I forgot to mention that I got very minor speedups in other runs that I did (the posted one is only the first one). Obviously, this is not the case for the posted results. |
|
@dwijnand As I mention in my comment, I did not intend to make a rigorous, repeatable, scientifically-provable benchmark that shows the real impact of this change. What I tried is to provide a faint measurable impact that people can use to discern whether this is a valuable change or not regarding performance and readability. This benchmark is missing a lot of stuff (GC logs, debug and optimization information, proper warm up, etc). Honestly, if I were to benchmark this seriously, I would use |
|
Don't get me wrong, I agree 1 datapoint is better than none. But for a project as established and important as zinc I would consider setting up a suite of JMH benchmarks to be a pre-requisite to working on and accepting any performance-related code changes. Otherwise we're kind of just shooting in the dark; just guessing at how reorganising the code impacts its performance when running on the JVM and by the JIT. |
|
I agree with you that such a suite is required, and I wish we already had that. It's true that Zinc is a very established project in the community but no performance fixes (e.g. https://github.com/sbt/zinc/pulls?q=is:pr+performance+is:closed and more issues in sbt/sbt) have been blocked by the need of such a suite. Even though I would really like to have it, I don't have the time to do it right now. The time I can invest on Zinc is fairly constrained -- I'm focusing on other use cases of the incremental compilation that are more significant to the community (correctness) so that we can have a beta soon; then I will focus on java compatibility and benchmarking. My main purpose while creating this PR was enhancing readability of the code and doing an algorithm that is more intuitive and easier to follow. I then realised that it would also be more performant, and that's why I chose the title of this PR. What I propose is that we merge this and when I have the time to do proper benchmarking, I will get back to it and prove that these changes don't affect performance (though the speedup in cold start is significant). |
|
Btw I would consider a JMH benchmark suite to have been a pre-requisite to those other performance-related changes too. |
|
@jvican thanks for the results - over 60% speedup on cold machine is really nice (but TBH it is strange that Dependecy phase in 1st compilation is so slow). @dwijnand I think having JMH would be really nice but no one working only partially (or in the free time) on zinc don't have time for that. E.g. in my case I just saw that compilation with zinc was way slower that was without so in order to use zinc I needed to fix that (and fixing took quite a while) so there was no time left for JMH. |
|
After sleeping on this, I've decided to implement a JMH benchmark suite for Zinc. I'll update this PR once it's done. I think that this is becoming even more key after the changes I've done to |
|
After some days of work, I already have a working implementation of JMH benchmarks. I'll update this PR with it and the results. I may also add more benchmarks. |
This commit introduces JMH benchmarks as requested by sbt#225. The strategy for the JMH benchmarks is the following: 1. Clone repository to be compiled and checkout at concrete commit (for reproducibility). 2. Generate sbt task for every subproject we want to test. This task will tell us information about the compiler options: sources and classpath (scalac options are integrated in next commit). 3. Execute the sbt tasks and get the output. 4. Instantiate a compiler fully integrated with the compiler bridge that has all the parameters given by the sbt output. 5. Run and enjoy. The JMH benchmarks run this for every iteration, for now. However, they only measure the running time of the **whole** compiler pipeline. There is no reliable way to do it only for our Zinc phases since they are tightly coupled to the compiler, so this is the best we can do. The JMH benchmarks are only 2.12 compatible. This commit introduces a benchmark for Shapeless and shows how easy it is to extend for other codebases. It also adds some tests to make sure that the required scaffolding works. For that, we have to modify the accessibility of some methods in our infrastructure, particularly `ScalaCompilerForUnitTesting`.
jvican
changed the title
|
Yes, this way we don't benchmark sbt. What I like about this approach is that it's pretty scalable, you can benchmark any project in the community that uses sbt and compiles with 2.12.1. In fact, I was thinking of running these results with |
|
Okay, here's some more info regarding memory consumption. Running JMH with gc profilerBeforeAfterThe following data confirms that this PR produces 20% less memory in old generation, as expected. This saves around 400ms in GC running time. Not a lot, but something. |
`zincBenchmarks` was a 2.12.x project because it was using the right-biased `Either`. This commit makes it compile with 2.11.8.
|
CI is green. |
|
Merged. Thanks for the contribution! |
That's pretty concerning. Have you done run with different values of |
|
I don't know how (or if) limiting |
Perhaps shapeless isn't a great example. Depending on which parts you're compiling, it might have proportionally more time spent in the typer phase than a typical project, so improvements to SBT's phases don't have as much impact. Having a mode in this benchmark tool to turn the SBT phases into no-ops (or remove them completely), would help give us a number that we could discuss easily (e.g "the overhead of SBT in a full build was reduced from 18% to 15%"). |
|
Performance of SBT's phases is worst in deep hierarchies, as it calls |
|
Finally, I'm a little bit nervous about making structural changes to Zinc in this codebase, because we sacrifice the ability to run the new version through the community build and check for crashes. Would it be feasible to take inspiration for 4a09b04 and make an sbt 0.13 plugin that augments the standard |
From now on, I'll use the Scala standard libray for those changes. In fact, my upcoming PR fixing ASF has such information in the commit messages.
This is a marvelous idea. Thanks, I would very much like to do this so that people can benefit automatically from the improvements in latest Zinc. |
|
Belated many thanks @jvican for bootstrapping JMH in zinc! (I'm still catching up) |
I've been thinking about how to do this, but it does not seem easy. Zinc 1.0 is compiled for 2.11+ and sbt runs in 2.10, so to enable this we'd need to have a client (sbt 0.13) and a server (Zinc 1.0) communicating via files/messages. The server would need to keep a cache of hot compilers too. Is this how you envisioned the sbt plugin when you proposed the idea or am I missing something? |
|
You're right, classloader isolation is the way to go. I forgot this is the way the Scala compiler is actually run by sbt. Thanks for the pointer @dwijnand! |
This commit introduces JMH benchmarks as requested by sbt#225. The strategy for the JMH benchmarks is the following: 1. Clone repository to be compiled and checkout at concrete commit (for reproducibility). 2. Generate sbt task for every subproject we want to test. This task will tell us information about the compiler options: sources and classpath (scalac options are integrated in next commit). 3. Execute the sbt tasks and get the output. 4. Instantiate a compiler fully integrated with the compiler bridge that has all the parameters given by the sbt output. 5. Run and enjoy. The JMH benchmarks run this for every iteration, for now. However, they only measure the running time of the **whole** compiler pipeline. There is no reliable way to do it only for our Zinc phases since they are tightly coupled to the compiler, so this is the best we can do. The JMH benchmarks are only 2.12 compatible. This commit introduces a benchmark for Shapeless and shows how easy it is to extend for other codebases. It also adds some tests to make sure that the required scaffolding works. For that, we have to modify the accessibility of some methods in our infrastructure, particularly `ScalaCompilerForUnitTesting`.
This commit introduces JMH benchmarks as requested by sbt/zinc#225. The strategy for the JMH benchmarks is the following: 1. Clone repository to be compiled and checkout at concrete commit (for reproducibility). 2. Generate sbt task for every subproject we want to test. This task will tell us information about the compiler options: sources and classpath (scalac options are integrated in next commit). 3. Execute the sbt tasks and get the output. 4. Instantiate a compiler fully integrated with the compiler bridge that has all the parameters given by the sbt output. 5. Run and enjoy. The JMH benchmarks run this for every iteration, for now. However, they only measure the running time of the **whole** compiler pipeline. There is no reliable way to do it only for our Zinc phases since they are tightly coupled to the compiler, so this is the best we can do. The JMH benchmarks are only 2.12 compatible. This commit introduces a benchmark for Shapeless and shows how easy it is to extend for other codebases. It also adds some tests to make sure that the required scaffolding works. For that, we have to modify the accessibility of some methods in our infrastructure, particularly `ScalaCompilerForUnitTesting`.
This commit introduces changes to Scala 2.10 sources from the following PRs: * sbt/zinc#225 * sbt/zinc#216 * sbt/zinc#206 * sbt/zinc#221 It also removes a stub for 2.8 compatibility in `DelegatingReporter`. Support for Scala 2.8 compatibility is not already maintained it.
This commit introduces JMH benchmarks as requested by sbt/zinc#225. The strategy for the JMH benchmarks is the following: 1. Clone repository to be compiled and checkout at concrete commit (for reproducibility). 2. Generate sbt task for every subproject we want to test. This task will tell us information about the compiler options: sources and classpath (scalac options are integrated in next commit). 3. Execute the sbt tasks and get the output. 4. Instantiate a compiler fully integrated with the compiler bridge that has all the parameters given by the sbt output. 5. Run and enjoy. The JMH benchmarks run this for every iteration, for now. However, they only measure the running time of the **whole** compiler pipeline. There is no reliable way to do it only for our Zinc phases since they are tightly coupled to the compiler, so this is the best we can do. The JMH benchmarks are only 2.12 compatible. This commit introduces a benchmark for Shapeless and shows how easy it is to extend for other codebases. It also adds some tests to make sure that the required scaffolding works. For that, we have to modify the accessibility of some methods in our infrastructure, particularly `ScalaCompilerForUnitTesting`. Rewritten from sbt/zinc@4a09b04
This commit introduces changes to Scala 2.10 sources from the following PRs: * sbt/zinc#225 * sbt/zinc#216 * sbt/zinc#206 * sbt/zinc#221 It also removes a stub for 2.8 compatibility in `DelegatingReporter`. Support for Scala 2.8 compatibility is not already maintained it. Rewritten from sbt/zinc@5d46c1b
This pull request applies some minor fixes and improves the performance of the
Dependencyphase by merging the two-step implementation into one step. For anexplanation of the applied changes and its effect to performance, please check
the commit messages.
This work is done by the Scala Center.