watchman doesn't like buck-out

[spearce]

I'm turning off watchman for now. Watchman doesn't like my buck-out directory and now tells me it will refuse to work until I go read a webpage.

buck and watchman should figure out how to use buck-out together without me reading about a poison inotify add watch.

Log:
Watchman threw an exception while parsing file changes.
A non-recoverable condition has triggered. Watchman needs your help!
The triggering condition was at timestamp=1408571228: inotify-add-watch(/home/sop/git/gerrit/buck-out/bin/gerrit-acceptance-tests/src/test/jad
All requests will continue to fail with this message until you resolve
the underlying problem. You will find more information on fixing this at
https://facebook.github.io/watchman/troubleshooting.html#poison-inotify-add-watch

[2014-08-20 14:51:44.277][error][command:0030deb7-d702-48f4-eb31-5a7e1f73bb29][tid:57][com.facebook.buck.util.WatchmanWatcher] Error in Watchman output
com.facebook.buck.util.WatchmanWatcherException: A non-recoverable condition has triggered. Watchman needs your help!
The triggering condition was at timestamp=1408571228: inotify-add-watch(/home/sop/git/gerrit/buck-out/bin/gerrit-acceptance-tests/src/test/java/com/google/gerrit/acceptance/rest/project/java_test_GarbageCollectionIT_tmp/gerrit_test_1205646360960983060/tmp) -> Permission denied
All requests will continue to fail with this message until you resolve
the underlying problem. You will find more information on fixing this at
https://facebook.github.io/watchman/troubleshooting.html#poison-inotify-add-watch

    at com.facebook.buck.util.WatchmanWatcher.postEvents(WatchmanWatcher.java:290)
    at com.facebook.buck.cli.Main$Daemon.watchFileSystem(Main.java:265)
    at com.facebook.buck.cli.Main$Daemon.access$1000(Main.java:146)
    at com.facebook.buck.cli.Main.getParserFromDaemon(Main.java:668)
    at com.facebook.buck.cli.Main.executeCommand(Main.java:563)
    at com.facebook.buck.cli.Main.runMainWithExitCode(Main.java:429)
    at com.facebook.buck.cli.Main.tryRunMainWithExitCode(Main.java:834)
    at com.facebook.buck.cli.Main.runMainThenExit(Main.java:858)
    at com.facebook.buck.cli.Main.nailMain(Main.java:881)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)

[Coneko]

This is the correct link:

https://facebook.github.io/watchman/docs/troubleshooting.html#poison-inotify-add-watch

Can you check the permissions on those files?

Could it be buck is trying to get watchman to watch a file on which it does not have read permission?

cc @wez

[spearce]

That's crap.

Buck started Watchman. Buck created buck-out. I didn't mess with it. Watchman hated on whatever Buck did inside of buck-out while Buck was trying to run a build for me.

[wez]

I understand your frustration, but we do need some information from you to help clarify what is going on; I think there are a couple of issues here:

1. your .watchmanconfig doesn't list buck-out in its ignore_dirs list. This should be the standard deployment for buck and we do need to make setting and discovering this easier and more automated
2. Something is creating dirs under buck-out that you don't have access to. Either the permissions are completely screwed up or perhaps you sudo ran something and left something behind. We'd like to understand which of these it is so that we can figure out if this is a bug that we need to fix, or a general problem with surfacing unexpected conditions
3. due to an oversight, watchman is treating this condition (I'm assuming EACCES) as a poison state when it shouldn't, leaving you with a stuck watchman process. I've pushed facebook/watchman@2a52837 as a speculative fix for this.

[spearce]

1. I didn't realize I need to write a .watchmanconfig in order to use buck. I've been using buck basically since it was released. Looks like we need to better message setting up .watchmanconfig. Maybe if it really is that important, buck should refuse to work until it can verify you have a correctly created .watchmanconfig that does the job of ignoring your buck-out directory since buck can't tell watchman to do that on its own.

2. That was a java_test() running from buck. The directory is long since gone. It wasn't a sudo ran something mistake. Its possible watchman couldn't access the directory because it came and went and the deletion event was missed.

3. Maybe it really is a poison state, because the test created the directory, then deleted the directory, and watchman didn't like how that turned out.

[wez]

For 1), this is on our radar; there are pieces available for buck to interrogate watchman to surface this, it just hasn't been done yet.

For 2), I suspect that either the access mode was broken (I've seen stuff get chmod'd to random bits in the past) or that the dir went away. The man page isn't clear on the error code reported in this case; we already had code to handle ENOENT because we've seen this in the wild, but this code isn't listed on my current man page. :-/

For 3), the intent of poison states is only to enter them if there is nothing that watchman can do itself to remedy things such that it can give you a sane outcome. It refuses all requests across all watches to ensure that you notice the issue. A dir that is not watchable doesn't warrant this level of response, but perhaps could be surfaced better in any case.

[davido]

Let's list buck-out in .watchmanconfig in its ignore_dirs then [1].

[1] https://gerrit-review.googlesource.com/59450

[wez]

@davido you should only need to specify ignore_dirs; the default ignore_vcs has the usual suspects

[BruceZu]

back-out should be the default value of ignore_dirs without need user config it explicitly.
/Bruce

Sent from my Sony Xperia™ smartphone

---- David Ostrovsky wrote ----

Let's list buck-out in .watchmanconfig in its ignore_dirs then [1].

[1] https://gerrit-review.googlesource.com/59450

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/183#issuecomment-52882175.

[shs96c]

Totally agreed.

[sunshowers]

Per a discussion @wez, @dreiss and I had, that's not really possible since Watchman is also used by tools other than Buck -- in other words, the Watchman process isn't owned by Buck in any sense. I believe the right approach is for Buck to ask Watchman what the ignore directories are and loudly complain when buck-out isn't in them.

[sdwilsh]

99a728c made it so we automatically have watchman ignore any paths in your [project] ignore_dirs section. I'm going to go ahead and make us also automatically not watch buck-out, and buck-cache.

[sdwilsh]

What would still be left to do is to have Buck warn/error when .watchmanconfig isn't configured correctly. @natthu, want to help with that?

[sdwilsh]

Even better, 53f2b60 made us ignore buck-out and buck-cache, so we just need to verify that the .watchmanconfig is correct, and surface any issues to the user.

[Coneko]

Closing, the commits @sdwilsh mentions should have fixed this.

Let us know if it happens again.

[sdwilsh]

To be clear, we don't need to verify that .watchmanconfig is correct, because Buck now tells watchman what it can safely ignore. You don't need a .watchmanconfig file unless you have other tools (like hg) that also use watchman.