EMFILE error when using fs.watch on many items #2479

TrevorBurnham opened this Issue Jan 6, 2012 · 21 comments

8 participants


Test case:

var fs = require('fs');

for (var i = 1; i <= 500; i++) {
  fs.writeFileSync("" + i + ".tmp", '');

for (i = 1; i <= 500; i++) {
  try {
    fs.watch("" + i + ".tmp", function() {});
  } catch (e) {

With Node 0.6.2 under OS X, I consistently get output of 250, meaning that 249 files were successfully watched, but an EMFILE error occurred when trying to watch the 250th. The pertinent part of the stack trace is

Error: watch EMFILE
    at errnoException (fs.js:605:11)
    at FSWatcher.start (fs.js:632:11)
    at Object.watch (fs.js:660:11)

Several folks have experienced the error when using CoffeeScript's watch mode, particularly in a directory that's a git repository (see CoffeeScript issue #1537).

Adding a 100ms interval between each file, I get the same result. So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit? Pinging @bnoordhuis.

Node.js Foundation member

So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit?

Yes, ulimit -n 16384. It's not a Node limitation, it's a per-process limit that's enforced by the operating system (presumably OS X in your case, that's the only OS I know of that defaults to 256).

@bnoordhuis bnoordhuis closed this Jan 6, 2012

Ah, thanks for the clarification.


I'm running node 0.6.9 on linux mint 12. ulimit -n returns 65536 so as far as I can see it's not a file handle limit. The above test fails at 116.

Replacing fs.watch with fs.watchFile works fine.

Node.js Foundation member

@wombleton: What does the strace log look like?

Error: watch EMFILE
    at errnoException (fs.js:636:11)
    at FSWatcher.start (fs.js:663:11)
    at Object.watch (fs.js:691:11)
Node.js Foundation member

Heh, that's not quite what I mean. Can you gist the output of strace node script.js? If you pass -o trace.log to strace, it'll write to a file instead of the console.


Ah! Now I know about strace, I had it pegged as a slightly strange contraction. Apologies.


Node.js Foundation member

You're running into a linux-specific system limit, namely the maximum number of inotify instances.

$ sysctl -a | grep fs.inotify
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.inotify.max_queued_events = 16384

It's something that we can work around. I've opened a libuv issue for it: joyent/libuv#300.

Node.js Foundation member

I'm using fs.watch on node 0.8.2 to watch a directory tree. I'm only watching the directories/subdirectories not each individual file.

This is working beatifully on Linux (Ubuntu 10.04). But we are hitting EMFILE error on Mac OS. It looks like this error condition is documented here:


as happening when

   [EMFILE]           The per-process descriptor table is full.

Our application does need to work on Mac OS as well as Unix and Windows. So I wonder... is there a way to increase this 'per-process descriptor table' size to something reasonable for our node process?

Any suggestions to work around this Mac OS limit welcome.


PS: I did try a couple of google searches to find the answer to my question, but the most useful result that turned up so far has been this issue :-)

Node.js Foundation member

@kdvolder Try e.g. ulimit -n 65536. With OS X you may have to log in again after that.


Further note: ulimit command as suggested above doesn't seem to exist on Mac OS. We tried using the equivalent Mac OS command but it didn't seem to work (my guess is because the limit we are hitting is not that of 'open files' but the 'The per-process descriptor table is full.' whatever that is, seems like it is maybe a different limit.


FYI: command we tried on OS X was this one:
sudo sysctl -w kern.maxfilesperproc=400000

Setting it however seemed not have any effect on the limit for fs.watch. Still hitting error at around entry number #249.


FYI: Decided to resolved this by not using fs.watch on Mac OS. When we detect Mac OS we switch to a polling mechanims. I'd much prefer to use fs.watch of course. But I have not found any way so far to raise the limit. (And even if I could find the magic sudo command, it would be a poor user experience if a user would have to run some 'fix my os' script as sudo before being able to use our stuff.

I do understand this isn't a node problem perse, but some type of hard OS limit and so perhaps not a reason to re-open this issue... But maybe there is something that can be done on the node implementation anyway (after all bnoordhus was able to do something to the inotify implementation of fs.watch that seems to work quite nicely on Linux. So I'm hoping someone might come along and do something similar for the Mac OS / kqueue implementation.


It would be good to have that limit handy under some process property, so we can be aware, that on current platform we have following descriptors limit, and do some workarounds.

Node.js Foundation member

@medikoo That won't happen because the file descriptor limit is a platform specific thing. Non-CRT Windows doesn't really have one, for example. You're free to write an add-on that wraps getrlimit(RLIMIT_NOFILE) though.


I had the same problem (256 file limit on Mac OS X) and neither of the above worked. However, I found a solution on Superuser.com: http://superuser.com/a/514049

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 4096' | sudo tee -a /etc/profile

After rebooting, everything worked fine.


Some time ago I came up with watch wrapper that uses fs.watch until descriptors limit is approached and then switches internally to fs.watchFile see: https://github.com/medikoo/fs2/blob/master/README.md#watchpath

It works really well on my side. I use it to watch thousands of files at same time on OSX.

@meaku meaku referenced this issue in peerigon/alamid Jan 15, 2013

Update to webpack 0.9.x #144

@sokra sokra referenced this issue in webpack/webpack-tests-example Mar 30, 2013

Too many open files #1

@cspotcode cspotcode referenced this issue in beefsack/node-hound Jul 21, 2013

Cannot watch large directories #4

@silver83 silver83 referenced this issue in dbashford/mimosa Aug 11, 2013

very high cpu usage #263

@mjg2203 mjg2203 pushed a commit to mjg2203/edx-platform-seas that referenced this issue Oct 31, 2013
@cpennington cpennington Batched coffee --watch compilation in smaller batches
This is to work around an OSx issue that causes EMFILE errors in the
default configuration.
Ref: nodejs/node-v0.x-archive#2479

A command is quite useful when checking who is consuming inotify resources:

find /proc/*/fd -type l -lname '*inotify' -print 2>/dev/null \
  | cut -d / -f 3 | (while read pid; do ps --no-headers -p $pid -o user,command; done) \
  | sort | uniq -c

Saw it at http://dev.nethserver.org/issues/2850


This problem was not easy to fix with macOS Sierra because of a few changes which make recent instructions no longer value, specifically ulimit commands would not fix the problem as the new setting would not be persisted. I wrote up instructions for what I did to fix it for good. Now a React Native project is able to build and run the server which uses the watch utility to monitor for changes. The error for this issue what I was seeing previously and with these commands I was able to make it work.


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment