Spot checking compiler performance

[gkossakowski]

Today I learned from @adriaanm that a recent version of the Scala compiler compiles Scala code at speed of 3300 LoC/s on a warm jvm. This is a great number! I thought the baseline is 700 LoC/s. Based on my observations of scala projects, I worry the value of optimizing the compiler performance is not delivered to users. I see builds at 200-300 LoC/s. People continue to have slow builds and are unaware they could have it compile much faster.

The idea: make it extremely easy to spot check whether one has a compilation performance problem. The decision is based on comparing the compilation speed observed on a code base to the known baseline.

I think scalac should either print out the compilation speed by default or have an option one flips to get that information. An integration with sbt would be nice. At the moment, sbt's output is:

[success] Total time: 73 s, completed Nov 8, 2017 11:08:32 PM

Is 73s a lot or not? Who knows. How if it was:

[success] Total time: 73 s (2300 LoC/s), completed Nov 8, 2017 11:08:32 PM

Oh yeah, clearly we're in the ballpark of the baseline. No need to dig.

In terms of implementation: scalac would need to know how many lines of code it compiles, excluding comments and whitespace. That is not readily available in the internals but i think parser could be easily extended to count lines of code. The first tactic that comes to my mind: record all lines where an Ident or Apply or Import or a declaration (def, val, class, etc.) is created and count these lines. I think this could be implemented in 2-3 days of work.

I believe this feature would be the single best move to bridge the gap between what a modern scalac can offer to its users and what users actually get every day.

[fommil]

Isn't this exactly what @jvican is working on?

[jvican]

Not really, implementing the changes in scalac to know how many lines of code it compiles, excluding comments and whitespace, is not in the scope of SCP-010.

[fommil]

I think I prefer SCP-010 to this idea, which is trying to be a silver bullet.

[jvican]

What I like about this idea is that it's a metric that makes Scala users aware of how slow the compiler is with a number that can be used to compare across different codebases. I see it as an enabler for SCP-010: if users notice that the compiler is only doing 200 LOC/s, then it's time for them to sit down and study the statistics and the reason why it's so slow.

I have interesting data to share on this topic and will be released next week.

[fommil]

a high level trigger, if you will 😄 ... maybe an (optional) minimum threshold could point to the documentation of your new tool to gather more info.