New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge name hashing branch that improves incremental compiler #1010

Closed
gkossakowski opened this Issue Nov 27, 2013 · 12 comments

Comments

Projects
None yet
5 participants
@gkossakowski
Contributor

gkossakowski commented Nov 27, 2013

This is a meta-issue that allows people to get quick overview of where we are with merging name-hashing branch back to sbt code base.

Name hashing has been merged to sbt and released with sbt 0.13.2-M1!

Current status

The name hashing branch consists of 6 major components/changesets:

  • more fine-grained logging which helps to understand why files get invalidated (#941)
  • an option to control incremental compiler's behavior related to macro definitions (#976)
  • new source dependency tracking logic that is based on tree walking. The tree walk extracts dependencies introduced through member reference and inheritance (#1002)
  • tracking of used names in Scala source files (#1013)
  • hashing of public definitions grouped by simple name (#1026)
  • changes to incremental compiler algorithm that makes use of all data described above and improve the invalidation logic (#1042)

Only once the last bullet is merged you'll be able to see (and test) changes to incremental compiler behavior.

Targeted sbt version

At the moment we are targeting shipping improved incremental compiler with sbt 0.13.2. It's realistic we can do that but obviously there's no commitment on our side that we'll make it. Plans can always change.

Why would I care?

If you have large enough Scala project of certain (quite common) structure then you might suffer from the problem that sbt's incremental compiler recompiles too much when you modify your project. A good example are Play projects. Here's comparison of incremental compiler behavior in different sbt versions for some sample Play project and typical code modification scenarios. As you can see, in some cases the improvement can be quite dramatic.

NOTE: Once name hashing is merged not only sbt will benefit. Zinc will see the same improvements. Scala IDE (Eclipse plugin) and IntelliJ embed zinc so they will have better incremental compilation as well.

What's the basic idea?

I wrote some documentation that explains current incremental compiler design and challenges related to incremental compilation in Scala. There's also a chapter that explains the basic idea behind the name hashing algorithm.

@ghost ghost assigned gkossakowski Nov 27, 2013

@retronym

This comment has been minimized.

Show comment
Hide comment
@retronym

retronym Nov 27, 2013

Member

(edited ticket to link your WIP branch for the curious)

Member

retronym commented Nov 27, 2013

(edited ticket to link your WIP branch for the curious)

@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Dec 5, 2013

Contributor

dundundun! the end is extremely nigh!

Contributor

adriaanm commented Dec 5, 2013

dundundun! the end is extremely nigh!

@gkossakowski

This comment has been minimized.

Show comment
Hide comment
@gkossakowski

gkossakowski Jan 6, 2014

Contributor

Hooray! The name hashing algorithm has been merged. Now we need a release of sbt and we can start testing it quite seriously.

I'll be working on a longer blog post which contain specific numbers on improvements this algorithm introduces. Stay tuned!

Contributor

gkossakowski commented Jan 6, 2014

Hooray! The name hashing algorithm has been merged. Now we need a release of sbt and we can start testing it quite seriously.

I'll be working on a longer blog post which contain specific numbers on improvements this algorithm introduces. Stay tuned!

@benmccann

This comment has been minimized.

Show comment
Hide comment
@benmccann

benmccann Jan 7, 2014

Contributor

Woohoo!!! Best commit ever!

Contributor

benmccann commented Jan 7, 2014

Woohoo!!! Best commit ever!

@retronym

This comment has been minimized.

Show comment
Hide comment
@retronym

retronym Jan 7, 2014

Member

Folks could already start battle testing against a SBT nightly, right?

Member

retronym commented Jan 7, 2014

Folks could already start battle testing against a SBT nightly, right?

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Jan 7, 2014

Member

Yeah, however with this AND the sbt-server stuff in launcher I think we need a Milestone release of 0.13.2 so we can test these two high-pri features.

Member

jsuereth commented Jan 7, 2014

Yeah, however with this AND the sbt-server stuff in launcher I think we need a Milestone release of 0.13.2 so we can test these two high-pri features.

@gkossakowski

This comment has been minimized.

Show comment
Hide comment
@gkossakowski

gkossakowski Jan 7, 2014

Contributor

👍 to milestone release. I don't know what's schedule for sbt-server but from my point of view we can cut as soon as possible.

Contributor

gkossakowski commented Jan 7, 2014

👍 to milestone release. I don't know what's schedule for sbt-server but from my point of view we can cut as soon as possible.

@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Jan 8, 2014

Contributor

Awesome work y'all! 🍰

On Tuesday, January 7, 2014, Grzegorz Kossakowski wrote:

[image: 👍] to milestone release. I don't know what's schedule for
sbt-server but from my point of view we can cut as soon as possible.


Reply to this email directly or view it on GitHubhttps://github.com/sbt/sbt/issues/1010#issuecomment-31752551
.

Contributor

adriaanm commented Jan 8, 2014

Awesome work y'all! 🍰

On Tuesday, January 7, 2014, Grzegorz Kossakowski wrote:

[image: 👍] to milestone release. I don't know what's schedule for
sbt-server but from my point of view we can cut as soon as possible.


Reply to this email directly or view it on GitHubhttps://github.com/sbt/sbt/issues/1010#issuecomment-31752551
.

@gkossakowski

This comment has been minimized.

Show comment
Hide comment
@gkossakowski

gkossakowski Feb 6, 2014

Contributor

The name hashing has been shipped with sbt 0.13.2-M1. Closing this ticket.

Contributor

gkossakowski commented Feb 6, 2014

The name hashing has been shipped with sbt 0.13.2-M1. Closing this ticket.

@benmccann

This comment has been minimized.

Show comment
Hide comment
@benmccann

benmccann Apr 21, 2014

Contributor

I was surprised to see reference to this in Scala 2.11.0 release notes. Will this only work with Scala 2.11.0 or could we also use it with Scala 2.10.3?

Contributor

benmccann commented Apr 21, 2014

I was surprised to see reference to this in Scala 2.11.0 release notes. Will this only work with Scala 2.11.0 or could we also use it with Scala 2.10.3?

@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Apr 21, 2014

Contributor

It'll work in 2.10, though some corner cases could only be fixed in 2.11

On Monday, April 21, 2014, Ben McCann notifications@github.com wrote:

I was surprised to see reference to this in Scala 2.11.0 release noteshttp://scala-lang.org/news/2014/04/21/release-notes-2.11.0.html.
Will this only work with Scala 2.11.0 or could we also use it with Scala
2.10.3?

Reply to this email directly or view it on GitHubhttps://github.com/sbt/sbt/issues/1010#issuecomment-40970351
.

JOIN US. REGISTER TODAY! http://www.scaladays.org/
Scala http://www.scaladays.org/
Days http://www.scaladays.org/
June 16th-18th, http://www.scaladays.org/
Berlin http://www.scaladays.org/

Contributor

adriaanm commented Apr 21, 2014

It'll work in 2.10, though some corner cases could only be fixed in 2.11

On Monday, April 21, 2014, Ben McCann notifications@github.com wrote:

I was surprised to see reference to this in Scala 2.11.0 release noteshttp://scala-lang.org/news/2014/04/21/release-notes-2.11.0.html.
Will this only work with Scala 2.11.0 or could we also use it with Scala
2.10.3?

Reply to this email directly or view it on GitHubhttps://github.com/sbt/sbt/issues/1010#issuecomment-40970351
.

JOIN US. REGISTER TODAY! http://www.scaladays.org/
Scala http://www.scaladays.org/
Days http://www.scaladays.org/
June 16th-18th, http://www.scaladays.org/
Berlin http://www.scaladays.org/

@gkossakowski

This comment has been minimized.

Show comment
Hide comment
@gkossakowski

gkossakowski Apr 21, 2014

Contributor

Specifically, incremental compilation (including name hashing) can be ruined by bugs in Scala compiler. If Scala compiler reports to incremental compiler with unstable names (e.g. for existential types) then you'll get spurious recompilations. We've seen number of such examples and such bugs were fixed during Scala 2.11 development.

Your best combo these days is: naming hashing turned + Scala 2.11.

Contributor

gkossakowski commented Apr 21, 2014

Specifically, incremental compilation (including name hashing) can be ruined by bugs in Scala compiler. If Scala compiler reports to incremental compiler with unstable names (e.g. for existential types) then you'll get spurious recompilations. We've seen number of such examples and such bugs were fixed during Scala 2.11 development.

Your best combo these days is: naming hashing turned + Scala 2.11.

asfgit pushed a commit to apache/spark that referenced this issue May 11, 2014

Enabled incremental build that comes with sbt 0.13.2
More info at. sbt/sbt#1010

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #525 from ScrapCodes/sbt-inc-opt and squashes the following commits:

ba8fa42 [Prashant Sharma] Enabled incremental build that comes with sbt 0.13.2
(cherry picked from commit 70bcdef)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>

asfgit pushed a commit to apache/spark that referenced this issue May 11, 2014

Enabled incremental build that comes with sbt 0.13.2
More info at. sbt/sbt#1010

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #525 from ScrapCodes/sbt-inc-opt and squashes the following commits:

ba8fa42 [Prashant Sharma] Enabled incremental build that comes with sbt 0.13.2

douglaz added a commit to douglaz/spark that referenced this issue Jun 5, 2014

Enabled incremental build that comes with sbt 0.13.2
More info at. sbt/sbt#1010

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #525 from ScrapCodes/sbt-inc-opt and squashes the following commits:

ba8fa42 [Prashant Sharma] Enabled incremental build that comes with sbt 0.13.2

pdeyhim added a commit to pdeyhim/spark-1 that referenced this issue Jun 25, 2014

Enabled incremental build that comes with sbt 0.13.2
More info at. sbt/sbt#1010

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #525 from ScrapCodes/sbt-inc-opt and squashes the following commits:

ba8fa42 [Prashant Sharma] Enabled incremental build that comes with sbt 0.13.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment