New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

~compile only watches some files #3695

Closed
blast-hardcheese opened this Issue Oct 30, 2017 · 14 comments

Comments

Projects
None yet
6 participants
@blast-hardcheese

blast-hardcheese commented Oct 30, 2017

Steps

Setup

mkdir -p project src/main/scala/foo
echo sbt.version=1.0.3 > project/build.properties
echo 'println("woo")' > src/main/scala/App.scala
echo 'println("woo")' > src/main/scala/foo/bar.scala

Launch sbt ~compile

$ sbt
sbt> ~compile

Change files

  1. Changes to src/main/scala/App.scala always trigger a recompile
  2. Changes to src/main/scala/foo/bar.scala never trigger a recompile
  3. Create a new file, src/main/scala/foo/baz.scala. Changes to baz.scala trigger a recompile until the next full sbt launch (close and open again).

problem

~ doesn't always track files in directories that should be covered by the file watcher

expectation

~ always tracks files in directories that are covered by the file watcher

notes

sbt version: 1.0.3

@blast-hardcheese

This comment has been minimized.

Show comment
Hide comment
@blast-hardcheese

blast-hardcheese Oct 30, 2017

@dwijnand This is the issue you requested of me from Gitter, please let me know if I can be of any help going forward

blast-hardcheese commented Oct 30, 2017

@dwijnand This is the issue you requested of me from Gitter, please let me know if I can be of any help going forward

@blast-hardcheese

This comment has been minimized.

Show comment
Hide comment
@blast-hardcheese

blast-hardcheese Oct 30, 2017

Possibly related to #3687

blast-hardcheese commented Oct 30, 2017

Possibly related to #3687

@dwijnand

This comment has been minimized.

Show comment
Hide comment
@dwijnand
Member

dwijnand commented Oct 31, 2017

Thanks @blast-hardcheese.

@dwijnand dwijnand self-assigned this Oct 31, 2017

@dwijnand dwijnand added this to the 1.0.4 milestone Oct 31, 2017

@OlegYch

This comment has been minimized.

Show comment
Hide comment
@OlegYch

OlegYch Nov 6, 2017

Contributor

for me ~ doesn't see changes in tests, but when i modify main source ~testOnly runs twice
same for ~compile though

Contributor

OlegYch commented Nov 6, 2017

for me ~ doesn't see changes in tests, but when i modify main source ~testOnly runs twice
same for ~compile though

@dwijnand

This comment has been minimized.

Show comment
Hide comment
@dwijnand

dwijnand Nov 16, 2017

Member

I can't reproduce this. I followed the given steps and it all works as intended on my box:

> ~compile
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("woo")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("woo")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 2 s, completed 16-Nov-2017 16:05:04
1. Waiting for source changes... (press enter to interrupt)
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("woo")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:05:19
2. Waiting for source changes... (press enter to interrupt)
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 1 s, completed 16-Nov-2017 16:05:30
3. Waiting for source changes... (press enter to interrupt)
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala (again)")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:05:36
4. Waiting for source changes... (press enter to interrupt)
[info] Compiling 3 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala (again)")
[error] ^
[error] /s/t-watches/src/main/scala/foo/baz.scala:1:1: expected class or object definition
[error] println("I've created foo/baz.scala")
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:05:55
5. Waiting for source changes... (press enter to interrupt)
[info] Compiling 3 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala (again)")
[error] ^
[error] /s/t-watches/src/main/scala/foo/baz.scala:1:1: expected class or object definition
[error] println("I've changed foo/baz.scala")
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:06:04
6. Waiting for source changes... (press enter to interrupt)

Going to switch to #3687, hopefully I can reproduce that.

Member

dwijnand commented Nov 16, 2017

I can't reproduce this. I followed the given steps and it all works as intended on my box:

> ~compile
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("woo")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("woo")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 2 s, completed 16-Nov-2017 16:05:04
1. Waiting for source changes... (press enter to interrupt)
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("woo")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:05:19
2. Waiting for source changes... (press enter to interrupt)
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 1 s, completed 16-Nov-2017 16:05:30
3. Waiting for source changes... (press enter to interrupt)
[info] Compiling 2 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala (again)")
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:05:36
4. Waiting for source changes... (press enter to interrupt)
[info] Compiling 3 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala (again)")
[error] ^
[error] /s/t-watches/src/main/scala/foo/baz.scala:1:1: expected class or object definition
[error] println("I've created foo/baz.scala")
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:05:55
5. Waiting for source changes... (press enter to interrupt)
[info] Compiling 3 Scala sources to /s/t-watches/target/scala-2.12/classes ...
[error] /s/t-watches/src/main/scala/App.scala:1:1: expected class or object definition
[error] println("I've changed App.scala")
[error] ^
[error] /s/t-watches/src/main/scala/foo/bar.scala:1:1: expected class or object definition
[error] println("I've changed foo/bar.scala (again)")
[error] ^
[error] /s/t-watches/src/main/scala/foo/baz.scala:1:1: expected class or object definition
[error] println("I've changed foo/baz.scala")
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 0 s, completed 16-Nov-2017 16:06:04
6. Waiting for source changes... (press enter to interrupt)

Going to switch to #3687, hopefully I can reproduce that.

@eed3si9n eed3si9n modified the milestones: 1.0.4, 1.0.5 Nov 16, 2017

@blast-hardcheese

This comment has been minimized.

Show comment
Hide comment
@blast-hardcheese

blast-hardcheese Nov 17, 2017

@dwijnand I was just similarly unable to reproduce this on a linux VM, despite being able to reproduce it exactly the same way on my Macbook Air.

Upgrading Java now to see if that helps on my mac
Update: Upgrading to java version 1.8.0_151 did not impact reproducibility on OSX

Mac specs

OS version

10.12.6 (16G1036)

Java version

java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

SBT launcher

Experiment 1. Standard, 1.0.0 from Homebrew
Experiment 2. paulp's launcher

CentOS specs

Java version

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

SBT launcher

paulp's launcher

blast-hardcheese commented Nov 17, 2017

@dwijnand I was just similarly unable to reproduce this on a linux VM, despite being able to reproduce it exactly the same way on my Macbook Air.

Upgrading Java now to see if that helps on my mac
Update: Upgrading to java version 1.8.0_151 did not impact reproducibility on OSX

Mac specs

OS version

10.12.6 (16G1036)

Java version

java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

SBT launcher

Experiment 1. Standard, 1.0.0 from Homebrew
Experiment 2. paulp's launcher

CentOS specs

Java version

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

SBT launcher

paulp's launcher

@blast-hardcheese

This comment has been minimized.

Show comment
Hide comment
@blast-hardcheese

blast-hardcheese Nov 18, 2017

I was able to reproduce on a completely different mac, so I'm wondering if this is an OSX issue. That computer was also running 10.12.6

blast-hardcheese commented Nov 18, 2017

I was able to reproduce on a completely different mac, so I'm wondering if this is an OSX issue. That computer was also running 10.12.6

@blast-hardcheese

This comment has been minimized.

Show comment
Hide comment
@blast-hardcheese

blast-hardcheese Nov 18, 2017

I bisected and ended up on b9a4d32. -Dsbt.watch.mode=nio is definitely slow, although it seems to behave as expected, both in this test example and the project I was initially having issues with

blast-hardcheese commented Nov 18, 2017

I bisected and ended up on b9a4d32. -Dsbt.watch.mode=nio is definitely slow, although it seems to behave as expected, both in this test example and the project I was initially having issues with

@blast-hardcheese

This comment has been minimized.

Show comment
Hide comment
@blast-hardcheese

blast-hardcheese Nov 18, 2017

I wonder, @dwijnand, can you repro with -Dsbt.watch.mode=polling? I've got a script to repro, switching between nio and polling gives consistent results: nio always works as expected, polling always fails.

blast-hardcheese commented Nov 18, 2017

I wonder, @dwijnand, can you repro with -Dsbt.watch.mode=polling? I've got a script to repro, switching between nio and polling gives consistent results: nio always works as expected, polling always fails.

@DylanArnold

This comment has been minimized.

Show comment
Hide comment
@DylanArnold

DylanArnold Nov 18, 2017

For what it's worth I tried to repro this issue on Ubuntu linux a week ago to try and find out if it was the same issue I was having with incremental compilation in the 1.0.x series.

I was unable to reproduce the problem using the original steps.

DylanArnold commented Nov 18, 2017

For what it's worth I tried to repro this issue on Ubuntu linux a week ago to try and find out if it was the same issue I was having with incremental compilation in the 1.0.x series.

I was unable to reproduce the problem using the original steps.

@eatkins

This comment has been minimized.

Show comment
Hide comment
@eatkins

eatkins Nov 28, 2017

Contributor

I added an sbt plugin that resolves this issue for me on my mac: https://github.com/swoval/MacOSXWatchService

Contributor

eatkins commented Nov 28, 2017

I added an sbt plugin that resolves this issue for me on my mac: https://github.com/swoval/MacOSXWatchService

@dwijnand

This comment has been minimized.

Show comment
Hide comment
@dwijnand

dwijnand Nov 29, 2017

Member

I expect this to be fixed with sbt/io#86 too.

Member

dwijnand commented Nov 29, 2017

I expect this to be fixed with sbt/io#86 too.

dwijnand added a commit to sbt/io that referenced this issue Nov 29, 2017

dwijnand added a commit to dwijnand/sbt-io that referenced this issue Nov 29, 2017

eatkins added a commit to eatkins/sbt that referenced this issue Dec 1, 2017

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This was both a resource leak and likely was
preventing some updates from being seen.

May be related to sbt#3775, sbt#3695

eatkins added a commit to eatkins/sbt that referenced this issue Dec 1, 2017

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This may have prevented some of the file updates
from being seen by the watch service. Moreover, because we lost the
reference to the original WatchService, there was no way to clean it up,
which was a resource leak.

May be related to sbt#3775, sbt#3695

eatkins added a commit to eatkins/sbt that referenced this issue Dec 1, 2017

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This may have prevented some of the file updates
from being seen by the watch service. Moreover, because we lost the
reference to the original WatchService, there was no way to clean it up,
which was a resource leak.

May be related to sbt#3775, sbt#3695

eatkins added a commit to eatkins/sbt that referenced this issue Dec 1, 2017

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This may have prevented some of the file updates
from being seen by the watch service. Moreover, because we lost the
reference to the original WatchService, there was no way to clean it up,
which was a resource leak.

May be related to sbt#3775, sbt#3695

eatkins added a commit to eatkins/sbt that referenced this issue Dec 3, 2017

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This may have prevented some of the file updates
from being seen by the watch service. Moreover, because we lost the
reference to the original WatchService, there was no way to clean it up,
which was a resource leak.

May be related to sbt#3775, sbt#3695

eatkins added a commit to eatkins/sbt that referenced this issue Dec 7, 2017

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This was both a resource leak and likely was
preventing some updates from being seen.

May be related to sbt#3775, sbt#3695

@eed3si9n eed3si9n removed the in progress label Dec 13, 2017

@dwijnand dwijnand modified the milestones: 1.1.1, 1.1.0 Dec 13, 2017

@dwijnand

This comment has been minimized.

Show comment
Hide comment
@dwijnand

dwijnand Dec 13, 2017

Member

Re-opening to track sbt/io needs releasing for inclusion in 1.1.0-RC2.

Member

dwijnand commented Dec 13, 2017

Re-opening to track sbt/io needs releasing for inclusion in 1.1.0-RC2.

@dwijnand dwijnand reopened this Dec 13, 2017

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Dec 16, 2017

Member

New io with the fix was merged in #3816

Member

eed3si9n commented Dec 16, 2017

New io with the fix was merged in #3816

@eed3si9n eed3si9n closed this Dec 16, 2017

eed3si9n added a commit to eed3si9n/sbt that referenced this issue Mar 9, 2018

Cache watch service
I noticed that my custom WatchService was never cleaned up by sbt and
realized that after every build we were making a new WatchService. At
the same time, we were reusing the WatchState from the previous run,
which was using the original WatchService. This was particularly
problematic because it prevented us from registering any paths with the
new watch service. This may have prevented some of the file updates
from being seen by the watch service. Moreover, because we lost the
reference to the original WatchService, there was no way to clean it up,
which was a resource leak.

May be related to sbt#3775, sbt#3695
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment