This repository has been archived by the owner. It is now read-only.

EMFILE error when using fs.watch on many items #2479

Closed
TrevorBurnham opened this Issue Jan 6, 2012 · 21 comments

Comments

Projects
None yet
8 participants
@TrevorBurnham

TrevorBurnham commented Jan 6, 2012

Test case:

var fs = require('fs');

for (var i = 1; i <= 500; i++) {
  fs.writeFileSync("" + i + ".tmp", '');
}

for (i = 1; i <= 500; i++) {
  try {
    fs.watch("" + i + ".tmp", function() {});
  } catch (e) {
    console.log(i);
    process.exit(1);
  }
}

With Node 0.6.2 under OS X, I consistently get output of 250, meaning that 249 files were successfully watched, but an EMFILE error occurred when trying to watch the 250th. The pertinent part of the stack trace is

Error: watch EMFILE
    at errnoException (fs.js:605:11)
    at FSWatcher.start (fs.js:632:11)
    at Object.watch (fs.js:660:11)

Several folks have experienced the error when using CoffeeScript's watch mode, particularly in a directory that's a git repository (see CoffeeScript issue #1537).

Adding a 100ms interval between each file, I get the same result. So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit? Pinging @bnoordhuis.

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Jan 6, 2012

Member

So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit?

Yes, ulimit -n 16384. It's not a Node limitation, it's a per-process limit that's enforced by the operating system (presumably OS X in your case, that's the only OS I know of that defaults to 256).

Member

bnoordhuis commented Jan 6, 2012

So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit?

Yes, ulimit -n 16384. It's not a Node limitation, it's a per-process limit that's enforced by the operating system (presumably OS X in your case, that's the only OS I know of that defaults to 256).

@bnoordhuis bnoordhuis closed this Jan 6, 2012

@TrevorBurnham

This comment has been minimized.

Show comment
Hide comment
@TrevorBurnham

TrevorBurnham Jan 7, 2012

Ah, thanks for the clarification.

TrevorBurnham commented Jan 7, 2012

Ah, thanks for the clarification.

@wombleton

This comment has been minimized.

Show comment
Hide comment
@wombleton

wombleton Jan 30, 2012

I'm running node 0.6.9 on linux mint 12. ulimit -n returns 65536 so as far as I can see it's not a file handle limit. The above test fails at 116.

Replacing fs.watch with fs.watchFile works fine.

wombleton commented Jan 30, 2012

I'm running node 0.6.9 on linux mint 12. ulimit -n returns 65536 so as far as I can see it's not a file handle limit. The above test fails at 116.

Replacing fs.watch with fs.watchFile works fine.

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Jan 31, 2012

Member

@wombleton: What does the strace log look like?

Member

bnoordhuis commented Jan 31, 2012

@wombleton: What does the strace log look like?

@wombleton

This comment has been minimized.

Show comment
Hide comment
@wombleton

wombleton Jan 31, 2012

Error: watch EMFILE
    at errnoException (fs.js:636:11)
    at FSWatcher.start (fs.js:663:11)
    at Object.watch (fs.js:691:11)

wombleton commented Jan 31, 2012

Error: watch EMFILE
    at errnoException (fs.js:636:11)
    at FSWatcher.start (fs.js:663:11)
    at Object.watch (fs.js:691:11)
@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Jan 31, 2012

Member

Heh, that's not quite what I mean. Can you gist the output of strace node script.js? If you pass -o trace.log to strace, it'll write to a file instead of the console.

Member

bnoordhuis commented Jan 31, 2012

Heh, that's not quite what I mean. Can you gist the output of strace node script.js? If you pass -o trace.log to strace, it'll write to a file instead of the console.

@wombleton

This comment has been minimized.

Show comment
Hide comment
@wombleton

wombleton Jan 31, 2012

Ah! Now I know about strace, I had it pegged as a slightly strange contraction. Apologies.

https://gist.github.com/1707965

wombleton commented Jan 31, 2012

Ah! Now I know about strace, I had it pegged as a slightly strange contraction. Apologies.

https://gist.github.com/1707965

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Jan 31, 2012

Member

You're running into a linux-specific system limit, namely the maximum number of inotify instances.

$ sysctl -a | grep fs.inotify
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.inotify.max_queued_events = 16384

It's something that we can work around. I've opened a libuv issue for it: joyent/libuv#300.

Member

bnoordhuis commented Jan 31, 2012

You're running into a linux-specific system limit, namely the maximum number of inotify instances.

$ sysctl -a | grep fs.inotify
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.inotify.max_queued_events = 16384

It's something that we can work around. I've opened a libuv issue for it: joyent/libuv#300.

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis
Member

bnoordhuis commented Feb 23, 2012

@kdvolder

This comment has been minimized.

Show comment
Hide comment
@kdvolder

kdvolder Jul 18, 2012

I'm using fs.watch on node 0.8.2 to watch a directory tree. I'm only watching the directories/subdirectories not each individual file.

This is working beatifully on Linux (Ubuntu 10.04). But we are hitting EMFILE error on Mac OS. It looks like this error condition is documented here:

https://developer.apple.com/library/mac/#documentation/Darwin/Reference/Manpages/man2/kqueue.2.html

as happening when

   [EMFILE]           The per-process descriptor table is full.

Our application does need to work on Mac OS as well as Unix and Windows. So I wonder... is there a way to increase this 'per-process descriptor table' size to something reasonable for our node process?

Any suggestions to work around this Mac OS limit welcome.

kdvolder commented Jul 18, 2012

I'm using fs.watch on node 0.8.2 to watch a directory tree. I'm only watching the directories/subdirectories not each individual file.

This is working beatifully on Linux (Ubuntu 10.04). But we are hitting EMFILE error on Mac OS. It looks like this error condition is documented here:

https://developer.apple.com/library/mac/#documentation/Darwin/Reference/Manpages/man2/kqueue.2.html

as happening when

   [EMFILE]           The per-process descriptor table is full.

Our application does need to work on Mac OS as well as Unix and Windows. So I wonder... is there a way to increase this 'per-process descriptor table' size to something reasonable for our node process?

Any suggestions to work around this Mac OS limit welcome.

@kdvolder

This comment has been minimized.

Show comment
Hide comment
@kdvolder

kdvolder Jul 18, 2012

PS: I did try a couple of google searches to find the answer to my question, but the most useful result that turned up so far has been this issue :-)

kdvolder commented Jul 18, 2012

PS: I did try a couple of google searches to find the answer to my question, but the most useful result that turned up so far has been this issue :-)

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Jul 18, 2012

Member

@kdvolder Try e.g. ulimit -n 65536. With OS X you may have to log in again after that.

Member

bnoordhuis commented Jul 18, 2012

@kdvolder Try e.g. ulimit -n 65536. With OS X you may have to log in again after that.

@kdvolder

This comment has been minimized.

Show comment
Hide comment
@kdvolder

kdvolder Jul 18, 2012

Further note: ulimit command as suggested above doesn't seem to exist on Mac OS. We tried using the equivalent Mac OS command but it didn't seem to work (my guess is because the limit we are hitting is not that of 'open files' but the 'The per-process descriptor table is full.' whatever that is, seems like it is maybe a different limit.

kdvolder commented Jul 18, 2012

Further note: ulimit command as suggested above doesn't seem to exist on Mac OS. We tried using the equivalent Mac OS command but it didn't seem to work (my guess is because the limit we are hitting is not that of 'open files' but the 'The per-process descriptor table is full.' whatever that is, seems like it is maybe a different limit.

@kdvolder

This comment has been minimized.

Show comment
Hide comment
@kdvolder

kdvolder Jul 18, 2012

FYI: command we tried on OS X was this one:
sudo sysctl -w kern.maxfilesperproc=400000

Setting it however seemed not have any effect on the limit for fs.watch. Still hitting error at around entry number #249.

kdvolder commented Jul 18, 2012

FYI: command we tried on OS X was this one:
sudo sysctl -w kern.maxfilesperproc=400000

Setting it however seemed not have any effect on the limit for fs.watch. Still hitting error at around entry number #249.

@kdvolder

This comment has been minimized.

Show comment
Hide comment
@kdvolder

kdvolder Jul 19, 2012

FYI: Decided to resolved this by not using fs.watch on Mac OS. When we detect Mac OS we switch to a polling mechanims. I'd much prefer to use fs.watch of course. But I have not found any way so far to raise the limit. (And even if I could find the magic sudo command, it would be a poor user experience if a user would have to run some 'fix my os' script as sudo before being able to use our stuff.

I do understand this isn't a node problem perse, but some type of hard OS limit and so perhaps not a reason to re-open this issue... But maybe there is something that can be done on the node implementation anyway (after all bnoordhus was able to do something to the inotify implementation of fs.watch that seems to work quite nicely on Linux. So I'm hoping someone might come along and do something similar for the Mac OS / kqueue implementation.

kdvolder commented Jul 19, 2012

FYI: Decided to resolved this by not using fs.watch on Mac OS. When we detect Mac OS we switch to a polling mechanims. I'd much prefer to use fs.watch of course. But I have not found any way so far to raise the limit. (And even if I could find the magic sudo command, it would be a poor user experience if a user would have to run some 'fix my os' script as sudo before being able to use our stuff.

I do understand this isn't a node problem perse, but some type of hard OS limit and so perhaps not a reason to re-open this issue... But maybe there is something that can be done on the node implementation anyway (after all bnoordhus was able to do something to the inotify implementation of fs.watch that seems to work quite nicely on Linux. So I'm hoping someone might come along and do something similar for the Mac OS / kqueue implementation.

@medikoo

This comment has been minimized.

Show comment
Hide comment
@medikoo

medikoo Jul 27, 2012

It would be good to have that limit handy under some process property, so we can be aware, that on current platform we have following descriptors limit, and do some workarounds.

medikoo commented Jul 27, 2012

It would be good to have that limit handy under some process property, so we can be aware, that on current platform we have following descriptors limit, and do some workarounds.

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Jul 27, 2012

Member

@medikoo That won't happen because the file descriptor limit is a platform specific thing. Non-CRT Windows doesn't really have one, for example. You're free to write an add-on that wraps getrlimit(RLIMIT_NOFILE) though.

Member

bnoordhuis commented Jul 27, 2012

@medikoo That won't happen because the file descriptor limit is a platform specific thing. Non-CRT Windows doesn't really have one, for example. You're free to write an add-on that wraps getrlimit(RLIMIT_NOFILE) though.

@thaustein

This comment has been minimized.

Show comment
Hide comment
@thaustein

thaustein Dec 21, 2012

I had the same problem (256 file limit on Mac OS X) and neither of the above worked. However, I found a solution on Superuser.com: http://superuser.com/a/514049

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 4096' | sudo tee -a /etc/profile

After rebooting, everything worked fine.

thaustein commented Dec 21, 2012

I had the same problem (256 file limit on Mac OS X) and neither of the above worked. However, I found a solution on Superuser.com: http://superuser.com/a/514049

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 4096' | sudo tee -a /etc/profile

After rebooting, everything worked fine.

@medikoo

This comment has been minimized.

Show comment
Hide comment
@medikoo

medikoo Dec 23, 2012

Some time ago I came up with watch wrapper that uses fs.watch until descriptors limit is approached and then switches internally to fs.watchFile see: https://github.com/medikoo/fs2/blob/master/README.md#watchpath

It works really well on my side. I use it to watch thousands of files at same time on OSX.

medikoo commented Dec 23, 2012

Some time ago I came up with watch wrapper that uses fs.watch until descriptors limit is approached and then switches internally to fs.watchFile see: https://github.com/medikoo/fs2/blob/master/README.md#watchpath

It works really well on my side. I use it to watch thousands of files at same time on OSX.

@sokra sokra referenced this issue Mar 30, 2013

Closed

Too many open files #1

mjg2203 pushed a commit to mjg2203/edx-platform-seas that referenced this issue Oct 31, 2013

Batched coffee --watch compilation in smaller batches
This is to work around an OSx issue that causes EMFILE errors in the
default configuration.
Ref: nodejs/node-v0.x-archive#2479
@liukun

This comment has been minimized.

Show comment
Hide comment
@liukun

liukun Aug 19, 2015

A command is quite useful when checking who is consuming inotify resources:

find /proc/*/fd -type l -lname '*inotify' -print 2>/dev/null \
  | cut -d / -f 3 | (while read pid; do ps --no-headers -p $pid -o user,command; done) \
  | sort | uniq -c

Saw it at http://dev.nethserver.org/issues/2850

liukun commented Aug 19, 2015

A command is quite useful when checking who is consuming inotify resources:

find /proc/*/fd -type l -lname '*inotify' -print 2>/dev/null \
  | cut -d / -f 3 | (while read pid; do ps --no-headers -p $pid -o user,command; done) \
  | sort | uniq -c

Saw it at http://dev.nethserver.org/issues/2850

@brennanMKE

This comment has been minimized.

Show comment
Hide comment
@brennanMKE

brennanMKE Oct 4, 2016

This problem was not easy to fix with macOS Sierra because of a few changes which make recent instructions no longer value, specifically ulimit commands would not fix the problem as the new setting would not be persisted. I wrote up instructions for what I did to fix it for good. Now a React Native project is able to build and run the server which uses the watch utility to monitor for changes. The error for this issue what I was seeing previously and with these commands I was able to make it work.

https://gist.github.com/brennanMKE/f6aa55b452ecda2f4c7a379e21647c88

brennanMKE commented Oct 4, 2016

This problem was not easy to fix with macOS Sierra because of a few changes which make recent instructions no longer value, specifically ulimit commands would not fix the problem as the new setting would not be persisted. I wrote up instructions for what I did to fix it for good. Now a React Native project is able to build and run the server which uses the watch utility to monitor for changes. The error for this issue what I was seeing previously and with these commands I was able to make it work.

https://gist.github.com/brennanMKE/f6aa55b452ecda2f4c7a379e21647c88

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.