Merge name hashing branch that improves incremental compiler

[gkossakowski]

This is a meta-issue that allows people to get quick overview of where we are with merging name-hashing branch back to sbt code base.

Name hashing has been merged to sbt and released with sbt 0.13.2-M1!

Current status

The name hashing branch consists of 6 major components/changesets:

• more fine-grained logging which helps to understand why files get invalidated (More detailed logging and APIChanges refactoring #941)
• an option to control incremental compiler's behavior related to macro definitions (Make recompilation on macro definition optional #976)
• new source dependency tracking logic that is based on tree walking. The tree walk extracts dependencies introduced through member reference and inheritance (Extract dependencies by tree walking #1002)
• tracking of used names in Scala source files (Used names extraction logic #1013)
• hashing of public definitions grouped by simple name (Add hashing of public names defined in a source file. #1026)
• changes to incremental compiler algorithm that makes use of all data described above and improve the invalidation logic (Implement name hashing algorithm in incremental compiler #1042)

Only once the last bullet is merged you'll be able to see (and test) changes to incremental compiler behavior.

Targeted sbt version

At the moment we are targeting shipping improved incremental compiler with sbt 0.13.2. It's realistic we can do that but obviously there's no commitment on our side that we'll make it. Plans can always change.

Why would I care?

If you have large enough Scala project of certain (quite common) structure then you might suffer from the problem that sbt's incremental compiler recompiles too much when you modify your project. A good example are Play projects. Here's comparison of incremental compiler behavior in different sbt versions for some sample Play project and typical code modification scenarios. As you can see, in some cases the improvement can be quite dramatic.

NOTE: Once name hashing is merged not only sbt will benefit. Zinc will see the same improvements. Scala IDE (Eclipse plugin) and IntelliJ embed zinc so they will have better incremental compilation as well.

What's the basic idea?

I wrote some documentation that explains current incremental compiler design and challenges related to incremental compilation in Scala. There's also a chapter that explains the basic idea behind the name hashing algorithm.

[retronym]

(edited ticket to link your WIP branch for the curious)

[adriaanm]

dundundun! the end is extremely nigh!

[gkossakowski]

Hooray! The name hashing algorithm has been merged. Now we need a release of sbt and we can start testing it quite seriously.

I'll be working on a longer blog post which contain specific numbers on improvements this algorithm introduces. Stay tuned!

[benmccann]

Woohoo!!! Best commit ever!

[retronym]

Folks could already start battle testing against a SBT nightly, right?

[jsuereth]

Yeah, however with this AND the sbt-server stuff in launcher I think we need a Milestone release of 0.13.2 so we can test these two high-pri features.

[gkossakowski]

👍 to milestone release. I don't know what's schedule for sbt-server but from my point of view we can cut as soon as possible.

[adriaanm]

Awesome work y'all! 🍰

On Tuesday, January 7, 2014, Grzegorz Kossakowski wrote:

[image: 👍] to milestone release. I don't know what's schedule for
sbt-server but from my point of view we can cut as soon as possible.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1010#issuecomment-31752551
.

[gkossakowski]

The name hashing has been shipped with sbt 0.13.2-M1. Closing this ticket.

[benmccann]

I was surprised to see reference to this in Scala 2.11.0 release notes. Will this only work with Scala 2.11.0 or could we also use it with Scala 2.10.3?

[adriaanm]

It'll work in 2.10, though some corner cases could only be fixed in 2.11

On Monday, April 21, 2014, Ben McCann notifications@github.com wrote:

I was surprised to see reference to this in Scala 2.11.0 release noteshttp://scala-lang.org/news/2014/04/21/release-notes-2.11.0.html.
Will this only work with Scala 2.11.0 or could we also use it with Scala
2.10.3?

Reply to this email directly or view it on GitHubhttps://github.com//issues/1010#issuecomment-40970351
.

JOIN US. REGISTER TODAY! http://www.scaladays.org/
Scala http://www.scaladays.org/
Days http://www.scaladays.org/
June 16th-18th, http://www.scaladays.org/
Berlin http://www.scaladays.org/

[gkossakowski]

Specifically, incremental compilation (including name hashing) can be ruined by bugs in Scala compiler. If Scala compiler reports to incremental compiler with unstable names (e.g. for existential types) then you'll get spurious recompilations. We've seen number of such examples and such bugs were fixed during Scala 2.11 development.

Your best combo these days is: naming hashing turned + Scala 2.11.