Large number of watchers starts failing hard on OSX

[ronkorving]

We are running a fairly large project with 922 JavaScript and .less files being watched by Webpack's watch function on OS X (10.10.2) running on Node.js 5.2.0. During the initial build, or right when it's done with that, the following happens:

2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamScheduleWithRunLoop: failed to create the cffd
2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamCreate: _FSEventStreamCreate: ERROR: could not open kqueue
2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamScheduleWithRunLoop(): failed assertion 'streamRef != NULL'

2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamStart(): failed assertion 'streamRef != NULL'

2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamStop(): failed assertion 'streamRef != NULL'

2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamUnscheduleFromRunLoop(): failed assertion 'streamRef != NULL'

2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamInvalidate(): failed assertion 'streamRef != NULL'

2015-12-10 18:01 node[76106] (FSEvents.framework) FSEventStreamRelease(): failed assertion 'streamRef != NULL'

I wonder if this may be related to #387 and we are simply hitting maximum open files limitations. Any other ideas I'm also open to, because this is a massive blocker in our development process and is affecting multiple people at once.

[saghul]

Did you raise the open file descriptor limit (see ulimit)? (Node used to do it automagically, but I'm not sure if that is the case now)

[saghul]

/cc @indutny

[ronkorving]

ulimit currently says unlimited, but I'm not sure I can trust that. I'm going to try and pull #641 and roll a Node.js build on top of that, see if the problem goes away.

[saghul]

Do a ulimit -a and check for "open files", or do uname -n.

[ronkorving]

Thanks (I feel like a noob now). 1000 is the answer. And we're watching 922 files. Add some other running apps to that, and it's likely we're hitting a limit there. Strangely enough, another coworker's limit is at 256 and it doesn't blow up....

[ronkorving]

If #641 solves it (I should know soon enough), I hope that can make it into the next release.

[ronkorving]

I compiled and tried to run it, but the output is exactly the same, which is really odd because the error (2nd line) mentions kqueue, while the commit message in #641 says it should not be using kqueue at all anymore.

[saghul]

@ronkorving note that the error comes from the OSX framework itself. kqueue does work with file fds, so FSEvents is probably using kqueue underneath and running into the fd limit anyway. Did raising the limit with ulimit -n 10000 avoid the issue?

[ronkorving]

The office has closed, so I will have to try with someone tomorrow. We tried to bump it, but it wouldn't let us... I'm sure there's a way, but I don't know that way off the top of my head. Will have to postpone until tomorrow I'm afraid.

[saghul]

No worries.

[indutny]

@ronkorving this is unfixable, OS X has a global OS limit. Sorry!

[indutny]

Ah, wait... This is something different. (Gosh premature commenting)

[indutny]

@ronkorving seems to be ulimit -n problem.

[ronkorving]

We upped the ulimit from 1000 to 3000, and the same errors appear. So it seems unrelated to this error.

@indutny any idea what that global OS limit is? (is it lower than ulimit can communicate it to be?)

[saghul]

What does your code do? Use fs.watch on a thousand different files?

Can you come up with a test case which consists just of core modules?
On Dec 11, 2015 02:10, "Ron Korving" notifications@github.com wrote:

We doubled the ulimit to 2000, and the same errors appear. So it seems
unrelated to this error.

—
Reply to this email directly or view it on GitHub
#645 (comment).

[ronkorving]

I could give it a shot, but right now (for the sake of the project) I'm focused on non-watch fast build solutions in Webpack (which, thank god, exist). What is happening behind the scenes in Webpack exactly, I don't know, but I can look into that and get back to you.

[ronkorving]

It seems it's using chokidar behind the scenes. You may be familiar with it?

source: https://github.com/webpack/watchpack/blob/master/lib/DirectoryWatcher.js#L7

[saghul]

No, I'm not familiar with that. I see it uses fs.watch* and fsevents, but I see nothing we can fix from libuv.

[saghul]

We upped the ulimit from 1000 to 3000, and the same errors appear. So it seems unrelated to this error.

Note that the limit applies to the current shell, so if you run ulimit in a shell and start node in another, that limit won't apply.

[ronkorving]

We ran the whole process in the shell where limit was raised 3000 (actually, it was 3000 through .bash_profile, and fresh shells were used).

In any case, I guess it really is out of the hands of libuv... I'll close the issue and pursue my dreams elsewhere ;) I made an issue over at the Webpack project. We'll see if we can make any progress going from that angle.

[saghul]

Okidoki! Just to discard culprints, you could try the following: create a test which creates a directory and 1000 files insise. Then create a watcher for each with fs.watchFile, and see what happens.

[saghul]

Also, make sure you use a Node version which has libuv >= 1.7.0, which includes this patch: 7980f13

[ronkorving]

I used latest Node with #641 merged into it, and the problem didn't go away, so I guess that patch was also a part of it. Over the weekend I explored an alternative for Webpack builds that gives us the same speed, but doesn't require watchers (build-on-demand with caching), so the team is happy again.

I'll create the test you ask for and see if I can replicate the issue though.

[ronkorving]

Well, I wrote a little test that watches thousands of files and folders, and can't get it to blow up... Not even with stock Node :-/

[saghul]

@ronkorving Oh. I'll tentatively close this then, since there is nothing we can do. If you manage to narrow it down, leave a comment and we can reopen it.

[ronkorving]

Sounds good.