Error when watching new files with BSD adapter (kqueue)

[ghost]

hi,

with #292 we have *BSD support again, which is really great,
thanks a lot :-)

I encounter a very rare issue, only using guard, when a new file will
fail to be watched if it is deleted quickly. I tried to solve it with
the ignore mechanism and increasing latency or wait_for_delay,
without success.

E, [2015-03-14T20:07:52.726600 #2299] ERROR -- : run() in thread failed: No such file or directory - Failed to open file /.../src/my/listen-kqueue/tmp/toto: File doesn't exist.:/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/watcher/file.rb:19:in `initialize'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/queue.rb:227:in `new'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/queue.rb:227:in `watch_file'
/.../src/sys/listen/lib/listen/adapter/bsd.rb:96:in `_watch_file'
/.../src/sys/listen/lib/listen/adapter/bsd.rb:89:in `block in _watch_for_new_file'
/usr/local/lib/ruby/2.2/find.rb:48:in `block (2 levels) in find'
/usr/local/lib/ruby/2.2/find.rb:47:in `catch'
/usr/local/lib/ruby/2.2/find.rb:47:in `block in find'
/usr/local/lib/ruby/2.2/find.rb:42:in `each'
/usr/local/lib/ruby/2.2/find.rb:42:in `find'
/.../src/sys/listen/lib/listen/adapter/bsd.rb:103:in `_find'
/.../src/sys/listen/lib/listen/adapter/bsd.rb:87:in `_watch_for_new_file'
/.../src/sys/listen/lib/listen/adapter/bsd.rb:68:in `_process_event'
/.../src/sys/listen/lib/listen/adapter/base.rb:47:in `block (2 levels) in configure'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/watcher.rb:90:in `call'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/watcher.rb:90:in `callback!'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/event.rb:80:in `callback!'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/queue.rb:337:in `block in process'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/queue.rb:337:in `each'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/queue.rb:337:in `process'
/.../.gem/ruby/22/gems/rb-kqueue-0.2.3/lib/rb-kqueue/queue.rb:316:in `run'
/.../src/sys/listen/lib/listen/adapter/bsd.rb:50:in `_run'
/.../src/sys/listen/lib/listen/adapter/base.rb:58:in `block in start'
/.../src/sys/listen/lib/listen/internals/thread_pool.rb:7:in `call'
/.../src/sys/listen/lib/listen/internals/thread_pool.rb:7:in `block in add'

While it often happens with VCS or editors, it can also happen
infrequently with simpler and valid use cases if the system is loaded
with many IO operations at the time. My understanding is that the
ignore mechanism only works later, preventing callbacks, after an
error could happen in the adapter. But the BSD adapter needs to search
for new files (walking the filesystem) after receiving directory
change events. It can take some time and fail if the file disappear,
I'm not sure if ignore or timing settings are designed to operate at
this level or circumvent this specific case?

I'm using a workaround (tjouan/listen@55dda3921dfb852e3d0ad035c1bff2cad7210309) since a few
weeks, works well for my use case with guard. Unfortunately, I was not
able to reproduce this in a deterministic way in isolation only with
listen, I'm not sure it's the proper fix, how to add tests for such a
change and if logger usage is correct.

If more info, other changes or a PR would help to solve this, let me
know.

[e2]

@tjouan - the BSD adapter is pretty specific and so is this issue - every other adapter watches directories instead of files (sides steps a lot of problems)

The fix looks fine and I can merge it.

Ideally there would be some unit test for the BSD adapter's logic - just for the sake of coverage and possibly simulating edge cases. I use Linux only and no one out there really uses BSD AFAIK, so if anyone does, any specs that test the adapter's logic would be nice (check out the darwin_spec.rb for an extreme example - I broke that adapter a few times since I don't have OSX, so I wrote generic unit tests that run everywhere).

But since few people use BSD, it's no big deal - just create a PR and let me know if you need a release (e.g. if you need this fix on multiple machines, you need to avoid forks, etc.). I can't test the fix, of course.

[ghost]

@e2 thanks, you confirm what I was suspecting about other adapters.

I agree for test coverage, I initially tried to add one for this
specific change, but the intense mocking and lengthy test setup I
started to write didn't feel right. I wanted to first start by adding
some more basic test coverage but lacked time. I will try to find some
in the future and work on this (after checking the darwin adapter
specs), as I understand the BSD adapter don't have many users :-)

I'll create the PR with previously linked commit in a moment, I
personally don't need a release as I use listen only on dev machines
with guard.

Thanks for the fast answer and offering a release!

Also, about the test suite, I was able to run it a few weeks ago (both
without or with my change), but right now I'm not able to install a
test dependency, update_rspec custom branch of celluloid fork not
readable:

$ bundle
Fetching git://github.com/chronomantic/celluloid.git
fatal: ambiguous argument 'update_rspec': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Git error: command `git rev-parse update_rspec` in directory /.../.gem/ruby/22/cache/bundler/git/celluloid-7c27201d98730b8e5dc4874dc6c3d624306dc4e8 has failed.
If this error persists you could try removing the cache directory '/.../.gem/ruby/22/cache/bundler/git/celluloid-7c27201d98730b8e5dc4874dc6c3d624306dc4e8'

Let me know if you want me to open new issue for this.

[e2]

@tjouan - I temporarily switched branches because I needed a patched version of Celluloid to work with. It's fixed now (using Celluloid master).

[e2]

@tjouan - don't worry about specs. Just that every once in a while I need to modify something in all the adapters - and it's too easy to break BSD (OSX and Linux have good-enough coverage, Windows is pretty generic, but BSD is "works until someone reports it never did work").

[ghost]

@e2, I was able to install and run tests again successfully, thanks.

About specs, I recognize the value, and it could also help people like
me who can hesitate to propose change, since as you say it can break
something easily. Next time I have an issue and want to experiment
things I'll check darwin adapter specs and give it a try :-)

Thanks for your work on listen!

[ghost]

For future reference, solved with: #301