Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overflows #155

merged 3 commits into from May 27, 2018


None yet
2 participants
Copy link

commented May 16, 2018

I am optimistic that these changes will stop the spurious test failures that we've seen. I was working on top of #151 because it added some additional synchronization that I needed.

@eatkins eatkins force-pushed the eatkins:overflows branch from c3b1452 to 3c15122 May 16, 2018

@eatkins eatkins force-pushed the eatkins:overflows branch 2 times, most recently from 3c15122 to 2699a84 May 18, 2018

Copy link

left a comment


@@ -146,11 +155,62 @@ private[sbt] object EventMonitor {
updatedFiles ++ newFiles ++ notExist

This comment has been minimized.

Copy link

eed3si9n May 19, 2018


@eatkins Thanks for the descriptive commit message on this commit.

Ethan Atkins added some commits May 15, 2018

Ethan Atkins
Handle overflows in EventMonitor
While working on a related project, I realized that it's essential to
handle the OVERFLOW case for a WatchEvent. The EventMonitor relies on
subdirectory creation events to create a new watch key for each
subdirectory. If those events are missed due to overflow, the
EventMonitor will not be monitoring the newly created subdirectories. To
fix this, when the EventMonitor detects an overflow, it will now poll
the directory of the watch key repeatedly until it stabilizes. Once that
happens, it will register all of the newly found directories (if any)
with the watch service and will create events for all of the files in
said directories.

It is still possible that a trigger could be missed in the event that a
lot of files in a watched directory cause an overflow before a valid
source file is touched and before the EventMonitor handles the overflow.
I am not particularly worried about this.

I also discovered that the PollingWatchService was using an unbounded
list of events. This seemed risky so I switched to using an
ArrayBlockingQueue of size 256, which is what the MacOSXWatchService
uses as well. Since the queue is now bounded, I added overflow events to
the PollingWatchService as well.

I also added synchronization for a few methods in PollingWatchey and
MacOSXWatchKey. This is more consistent with the AbstractWatchKey in the
jdk. The other methods return constant values so synchronization is

I ran a number of travis builds against the content of this commit and
none of them failed.
Ethan Atkins
Switch to Files.walkFileTree in PathFinder
The handleFileDescendant method was recursively constructing a list of
results by calling File.list and recursively calling back into itself
for each directory that it found in the listed files. There were two
performance bugs in this approach. One was that handleFileDescendant
actually called File.list twice for each directory. The other was that
after it got the second list of files, it had to filter the results
based on whether or not it was a directory. When I called jstack on my
sbt process during a slow test run, invariably one of the threads was
bloced on File.isDirectory.

The fix is simple, if verbose. Use Files.walkFileTree instead. It is
verbose, but it recursively lists all of the files in the tree specified
by the path. An alternative would be to use Files.walk. I didn't go this
route because Files.walk is somewhat hard to reason about because the
iterator can throw an exception at any time. This is a problem if the
directory is being traversed and modified at the same time. With
Files.walkFileTree, I know that if, for example, a directory disappears
from out from underneath us, then we'll just move on without throwing an
Ethan Atkins
Add synchronization
This test sometimes spuriously fails and I'm pretty sure it's because of
concurrent modifications of lines.

@eatkins eatkins force-pushed the eatkins:overflows branch from 2699a84 to 14f5e5d May 22, 2018

@eed3si9n eed3si9n merged commit 3328e02 into sbt:1.1.x May 27, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed

@eatkins eatkins deleted the eatkins:overflows branch Oct 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.