Spurious recompilation in multiproject builds with external subprojects.

[nuttycom]

I'm finding some strange behavior with the new external multiproject support. I have two projects, A and B which both dependOn external project C using RootProject("../c").

If I go to project A and compile it, C is also compiled, and all is well. However, if I then go to project B and compile it, this also causes C to be entirely recompiled. This of course is awful, because C takes a couple of minutes to compile and this happens whenever I switch between A and B.

Here is an example of build output related to the attached projects:

knuttycombe@floorshow sbt (master)]$ ls
a b c
[knuttycombe@floorshow sbt (master)]$ cd c/
[knuttycombe@floorshow c (master)]$ sbt compile
Getting net.java.dev.jna jna 3.2.3 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
1 artifacts copied, 0 already retrieved (838kB/80ms)
Getting Scala 2.8.1 (for sbt)...
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
4 artifacts copied, 0 already retrieved (15296kB/873ms)
Getting org.scala-tools.sbt sbt_2.8.1 0.10.0 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
34 artifacts copied, 0 already retrieved (6012kB/794ms)
[info] Set current project to default (in build file:/Users/knuttycombe/tmp/sbt/c/)
[info] Updating...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/c/target/scala-2.8.1.final/classes...
[success] Total time: 3 s, completed Jun 4, 2011 9:46:53 PM
[knuttycombe@floorshow c (master)]$ cd ../b
[knuttycombe@floorshow b (master)]$ sbt compile
Getting net.java.dev.jna jna 3.2.3 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
1 artifacts copied, 0 already retrieved (838kB/19ms)
Getting Scala 2.8.1 (for sbt)...
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
4 artifacts copied, 0 already retrieved (15296kB/248ms)
Getting org.scala-tools.sbt sbt_2.8.1 0.10.0 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
34 artifacts copied, 0 already retrieved (6012kB/63ms)
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/b/project/target/scala_2.8.1/classes...
[info] Set current project to b (in build file:/Users/knuttycombe/tmp/sbt/b/)
[info] Updating...
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/c/target/scala-2.8.1.final/classes...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/b/target/scala-2.8.1.final/classes...
[success] Total time: 3 s, completed Jun 4, 2011 9:47:16 PM
[knuttycombe@floorshow b (master)]$ cd ../a
[knuttycombe@floorshow a (master)]$ sbt compile
Getting net.java.dev.jna jna 3.2.3 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
1 artifacts copied, 0 already retrieved (838kB/17ms)
Getting Scala 2.8.1 (for sbt)...
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
4 artifacts copied, 0 already retrieved (15296kB/236ms)
Getting org.scala-tools.sbt sbt_2.8.1 0.10.0 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
34 artifacts copied, 0 already retrieved (6012kB/65ms)
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/a/project/target/scala_2.8.1/classes...
[info] Set current project to a (in build file:/Users/knuttycombe/tmp/sbt/a/)
[info] Updating...
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/c/target/scala-2.8.1.final/classes...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/a/target/scala-2.8.1.final/classes...
[success] Total time: 3 s, completed Jun 4, 2011 9:47:41 PM
[knuttycombe@floorshow a (master)]$ cd ../b/
[knuttycombe@floorshow b (master)]$ sbt compile
[info] Set current project to b (in build file:/Users/knuttycombe/tmp/sbt/b/)
[info] Compiling 1 Scala source to /Users/knuttycombe/tmp/sbt/c/target/scala-2.8.1.final/classes...
[success] Total time: 6 s, completed Jun 4, 2011 9:47:58 PM
[knuttycombe@floorshow b (master)]$

Something interesting to note: although c is recompiled when I switch back to b, the classes in b that depend on c are not!

[nuttycom]

Haven't been able to figure out how to attach a file for this ticket, so here's the source organization:

./a/project/Build.scala
import sbt._
import Keys._

object ABuild extends Build {
lazy val a = Project("a", file(".")) dependsOn (c)
lazy val c = RootProject(file("../c"))
}

./a/src/main/scala/a/A.scala
package a
import c.C

class A(c: C)

./b/project/Build.scala
import sbt._
import Keys._

object BBuild extends Build {
lazy val b = Project("b", file(".")) dependsOn (c)
lazy val c = RootProject(file("../c"))
}

./b/src/main/scala/b/B.scala
package b
import c.C

case class B(c: C)

./c/build.sbt
name := "c"

./c/src/main/scala/c/C.scala
package c

case class C(value: String)

[harrah]

Apparently, you are supposed to paste in a gist and link to it, which is somewhat unfortunate.

The cause of the problem is that by default, sbt uses the project/boot directory in the project you start sbt in and in particular, the Scala files there. So, when you move between projects, sbt sees a different version of scala-library.jar for c (the one in a/project/boot and then in b/project/boot) and thinks the source needs recompiling.

There are two solutions I can think of. First, you can have a global boot directory that is shared by all projects on your machine. You can do this by setting the sbt.boot.directory system property in your launcher script. It can also be passed directly to the sbt script to just try it out at first. For example:

sbt -Dsbt.boot.directory=/home/user/.sbt/boot/

The other option is to have another project that refers to the three projects and then always run sbt on that project. This other project doesn't have to contain the subdirectories.

[nuttycom]

Having a common SBT boot directory sounds like it might be generally good
practice in any case, because it would minimize startup times for new
projects as well. I'll give that a try, thanks!
On Jun 5, 2011 12:47 AM, "harrah" <
reply@reply.github.com>
wrote:

Apparently, you are supposed to paste in a gist and link to it, which is
somewhat unfortunate.

The cause of the problem is that by default, sbt uses the project/boot
directory in the project you start sbt in and in particular, the Scala files
there. So, when you move between projects, sbt sees a different version of
scala-library.jar for c (the one in a/project/boot and then in
b/project/boot) and thinks the source needs recompiling.

There are two solutions I can think of. First, you can have a global boot
directory that is shared by all projects on your machine. You can do this by
setting the sbt.boot.directory system property in your launcher script. It
can also be passed directly to the sbt script to just try it out at first.
For example:

sbt -Dsbt.boot.directory=/home/user/.sbt/boot/

The other option is to have another project that refers to the three
projects and then always run sbt on that project. This other project doesn't
have to contain the subdirectories.

Reply to this email directly or view it on GitHub:
harrah#35 (comment)

[nuttycom]

The global sbt boot directory looks like it does the trick. I still have a bunch more questions, but I'll bring them up on the list. Thanks!

[harrah]

To close this out, I added comments about this on Setup and Full Configuration.