Large number of node_modules/files causes "too many open files" on OS X #6952

Closed
emlai opened this Issue May 2, 2016 · 61 comments

Comments

Projects
None yet
@emlai

emlai commented May 2, 2016

_1 Upvote_ I have ~40000 files (~7GB altogether) in my public directory. When I run meteor, the "building the application" step takes 3 minutes. When I then press ctrlc to exit it, the following gets printed:

fs.js:439
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EMFILE, too many open files '/Users/emlai/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/static-assets/server/shell-server.js'
    at Object.fs.openSync (fs.js:439:18)
    at Object.fs.readFileSync (fs.js:290:15)
    at Object.Module._extensions..js (module.js:473:44)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /tools/runners/run-app.js:146:7
    at /Users/emlai/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:24:7
    at Array.forEach (native)
    at Function._.each._.forEach (/Users/emlai/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at /Users/emlai/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:23:7
    at Object.exports.noYieldsAllowed (/tools/utils/fiber-helpers.js:37:14)
    at runHandlers (/Users/emlai/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:20:16)
    at process.<anonymous> (/Users/emlai/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:32:5)
    at process.g (events.js:180:16)
    at process.emit (events.js:92:17)
    at Signal.wrap.onsignal (node.js:800:46)

My setup: Meteor 1.3.2.4 on OS X El Capitan 10.11.5 Beta.

» meteor list
accounts-facebook      1.0.9  Login service for Facebook accounts
accounts-password      1.1.8  Password support for accounts
accounts-ui            1.1.9  Simple templates to add login widgets to an app
blaze-html-templates   1.0.4  Compile HTML templates into reactive UI with Meteor Blaze
ecmascript             0.4.3  Compiler plugin that supports ES2015+ in all .js files
es5-shim               4.5.10  Shims and polyfills to improve ECMAScript 5 support
http                   1.1.5  Make HTTP calls to remote servers
jquery                 1.11.8  Manipulate the DOM using CSS selectors
meteor-base            1.0.4  Packages that every Meteor app needs
mobile-experience      1.0.4  Packages for a great mobile user experience
mongo                  1.1.7  Adaptor for using MongoDB and Minimongo over DDP
reactive-dict          1.1.7  Reactive dictionary
reactive-var           1.0.9  Reactive variable
sergeyt:typeahead      0.11.1_8  Autocomplete package for meteor powered by twitter typ...
standard-minifier-css  1.0.6  Standard css minifier used with Meteor apps by default.
standard-minifier-js   1.0.6  Standard javascript minifiers used with Meteor apps by de...
tracker                1.0.13  Dependency tracker to allow reactive callbacks

@emlai emlai changed the title from "Building the application" takes 3 minutes and crashes with many files in public/ to Many files in public/: Building is very slow and ctrl-c causes crash May 2, 2016

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 2, 2016

Member

@emlai What's the structure of the files in your public directory? Just 40k files in the directory? Or are they organized into sub-directories?

What is the output of running launchctl limit maxfiles on your terminal? It should show what the user-limits are for your account.

Member

abernix commented May 2, 2016

@emlai What's the structure of the files in your public directory? Just 40k files in the directory? Or are they organized into sub-directories?

What is the output of running launchctl limit maxfiles on your terminal? It should show what the user-limits are for your account.

@emlai

This comment has been minimized.

Show comment
Hide comment
@emlai

emlai May 2, 2016

@abernix They're organized into ~200 subdirectories inside public. Inside those subdirectories, there are 1-3 directories, inside which the actual files are located.

» launchctl limit maxfiles
    maxfiles    256            unlimited  

emlai commented May 2, 2016

@abernix They're organized into ~200 subdirectories inside public. Inside those subdirectories, there are 1-3 directories, inside which the actual files are located.

» launchctl limit maxfiles
    maxfiles    256            unlimited  
@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 2, 2016

Member

@emlai Ok, that's pretty reasonably matrixing of your files. It's showing your soft limit is 256 but your hard limit is unlimited, however, I think unlimited by default is still only 10k files on El Cap. Out of curiosity, what happens if you raise your user limits waaaay up (temporarily):

sudo launchctl limit maxfiles 524288 524288

Then try Meteor again.

Once you get the results (good or bad), you can put them back to how they were with:

sudo launchctl limit maxfiles 256 unlimited

(These are normally controlled by launchd at boot time, so absolute worse case a reboot will put you back exactly as things were before)

Member

abernix commented May 2, 2016

@emlai Ok, that's pretty reasonably matrixing of your files. It's showing your soft limit is 256 but your hard limit is unlimited, however, I think unlimited by default is still only 10k files on El Cap. Out of curiosity, what happens if you raise your user limits waaaay up (temporarily):

sudo launchctl limit maxfiles 524288 524288

Then try Meteor again.

Once you get the results (good or bad), you can put them back to how they were with:

sudo launchctl limit maxfiles 256 unlimited

(These are normally controlled by launchd at boot time, so absolute worse case a reboot will put you back exactly as things were before)

@emlai

This comment has been minimized.

Show comment
Hide comment
@emlai

emlai May 2, 2016

Using sudo launchctl limit maxfiles 524288 524288 didn't change anything.

emlai commented May 2, 2016

Using sudo launchctl limit maxfiles 524288 524288 didn't change anything.

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 3, 2016

Member

Ok, the specifics of actually changing the file limit are a bit hairy in OS X El Cap but I can confirm that this breaks for me too. It's highly possible that this would be fixed if Meteor used graceful-fs instead of fs as I'm sure openSync is, well, not being graceful, but not sure if that will take care of Windows, or if putting 40000 files into the public/ directory is something permitted. 😁

Reproduction Repo: https://github.com/abernix/bunch-o-files

Basically, this is a new meteor project with this commands:

for i in {1..40000}; do openssl rand -base64 16 | md5 | head -c16 | sed -e 's/^\(\(...\)\(.\).*\)$/mkdir -p public\/\2\/\3\/ \&\& touch public\/\2\/\3\/\1/'; done | sh
Member

abernix commented May 3, 2016

Ok, the specifics of actually changing the file limit are a bit hairy in OS X El Cap but I can confirm that this breaks for me too. It's highly possible that this would be fixed if Meteor used graceful-fs instead of fs as I'm sure openSync is, well, not being graceful, but not sure if that will take care of Windows, or if putting 40000 files into the public/ directory is something permitted. 😁

Reproduction Repo: https://github.com/abernix/bunch-o-files

Basically, this is a new meteor project with this commands:

for i in {1..40000}; do openssl rand -base64 16 | md5 | head -c16 | sed -e 's/^\(\(...\)\(.\).*\)$/mkdir -p public\/\2\/\3\/ \&\& touch public\/\2\/\3\/\1/'; done | sh
@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 5, 2016

To reproduce this bug you can try meteorchief base repo https://github.com/themeteorchef/base
Just clone, install packages and run, nothing happens at localhost:3000 in browser, and when hit Ctrl+C:

fs.js:439
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EMFILE, too many open files '/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/static-assets/server/shell-server.js'
    at Object.fs.openSync (fs.js:439:18)
    at Object.fs.readFileSync (fs.js:290:15)
    at Object.Module._extensions..js (module.js:473:44)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /tools/runners/run-app.js:146:7
    at /Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:24:7
    at Array.forEach (native)
    at Function._.each._.forEach (/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at /Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:23:7
    at Object.exports.noYieldsAllowed (/tools/utils/fiber-helpers.js:37:14)
    at runHandlers (/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:20:16)
    at process.<anonymous> (/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:32:5)
    at process.g (events.js:180:16)
    at process.emit (events.js:92:17)
    at Signal.wrap.onsignal (node.js:800:46)

Mac OS El Captain 10.11.4, node 0.10.44, npm 2.15.0

focused commented May 5, 2016

To reproduce this bug you can try meteorchief base repo https://github.com/themeteorchef/base
Just clone, install packages and run, nothing happens at localhost:3000 in browser, and when hit Ctrl+C:

fs.js:439
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EMFILE, too many open files '/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/static-assets/server/shell-server.js'
    at Object.fs.openSync (fs.js:439:18)
    at Object.fs.readFileSync (fs.js:290:15)
    at Object.Module._extensions..js (module.js:473:44)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /tools/runners/run-app.js:146:7
    at /Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:24:7
    at Array.forEach (native)
    at Function._.each._.forEach (/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at /Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:23:7
    at Object.exports.noYieldsAllowed (/tools/utils/fiber-helpers.js:37:14)
    at runHandlers (/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:20:16)
    at process.<anonymous> (/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/tool-env/cleanup.js:32:5)
    at process.g (events.js:180:16)
    at process.emit (events.js:92:17)
    at Signal.wrap.onsignal (node.js:800:46)

Mac OS El Captain 10.11.4, node 0.10.44, npm 2.15.0

@queso

This comment has been minimized.

Show comment
Hide comment
@queso

queso May 5, 2016

Contributor

@focused: I actually tried and couldn't reproduce this by downloading base... I cloned the repo, did a meteor npm i, then did a meteor run. Compiled and ran fine on my MacBook 12".

Contributor

queso commented May 5, 2016

@focused: I actually tried and couldn't reproduce this by downloading base... I cloned the repo, did a meteor npm i, then did a meteor run. Compiled and ran fine on my MacBook 12".

@mateodelnorte

This comment has been minimized.

Show comment
Hide comment
@mateodelnorte

mateodelnorte May 5, 2016

This is happening to 5 members on my team, and began immediately after the 1.3 upgrade. We don't have nearly that many files our public directory (maybe 30 total), but we do now have a large amount of files nested under ./node_modules/.

This is happening to 5 members on my team, and began immediately after the 1.3 upgrade. We don't have nearly that many files our public directory (maybe 30 total), but we do now have a large amount of files nested under ./node_modules/.

@merlinpatt

This comment has been minimized.

Show comment
Hide comment
@merlinpatt

merlinpatt May 8, 2016

Collaborator

What's the purpose of these 40000 files?

Collaborator

merlinpatt commented May 8, 2016

What's the purpose of these 40000 files?

@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 8, 2016

@queso What is your OSX, npm, node and Meteor versions? What if you enter ulimit -n and launchctl limit maxfiles?

On my mac pro:

focused:base focused$ launchctl limit maxfiles
    maxfiles    256            unlimited
focused:base focused$ ulimit -n
1024
focused:base focused$ ulimit
unlimited

https://github.com/themeteorchef/base has many packages and I wait too long before Application tries to start. Maybe we can find a solution..

focused commented May 8, 2016

@queso What is your OSX, npm, node and Meteor versions? What if you enter ulimit -n and launchctl limit maxfiles?

On my mac pro:

focused:base focused$ launchctl limit maxfiles
    maxfiles    256            unlimited
focused:base focused$ ulimit -n
1024
focused:base focused$ ulimit
unlimited

https://github.com/themeteorchef/base has many packages and I wait too long before Application tries to start. Maybe we can find a solution..

@abernix abernix changed the title from Many files in public/: Building is very slow and ctrl-c causes crash to Large number of node_modules/files causes "too many open files" on OS X May 9, 2016

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 10, 2016

Member

Those experiencing this, try to update this issue with the output of this command, run from your project root. It will print the top 5 directories with the most files and should be super discreet. It's generally expected that your .meteor directory will have 50k+ but the others might be causing problems. (This will only work for *nix users). Just for some extra possibly-insightful stats on this.

find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5
Member

abernix commented May 10, 2016

Those experiencing this, try to update this issue with the output of this command, run from your project root. It will print the top 5 directories with the most files and should be super discreet. It's generally expected that your .meteor directory will have 50k+ but the others might be causing problems. (This will only work for *nix users). Just for some extra possibly-insightful stats on this.

find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5
@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 10, 2016

focused:base focused$ find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5
60831 node_modules
 609 .meteor
  34 imports
  22 .git
  10 client
focused:base focused$

focused commented May 10, 2016

focused:base focused$ find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5
60831 node_modules
 609 .meteor
  34 imports
  22 .git
  10 client
focused:base focused$
@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 13, 2016

Today I added Semantic-UI package https://github.com/Semantic-Org/Semantic-UI-Meteor to a small meteor project and... the same error!

=> App running at: http://localhost:3000/
^C
fs.js:439
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EMFILE, too many open files '/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/static-assets/server/shell-server.js'
    at Object.fs.openSync (fs.js:439:18)

...

focused commented May 13, 2016

Today I added Semantic-UI package https://github.com/Semantic-Org/Semantic-UI-Meteor to a small meteor project and... the same error!

=> App running at: http://localhost:3000/
^C
fs.js:439
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EMFILE, too many open files '/Users/focused/.meteor/packages/meteor-tool/.1.3.2_4.lbyo5v++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/tools/static-assets/server/shell-server.js'
    at Object.fs.openSync (fs.js:439:18)

...
@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 13, 2016

Please help! :(

When my project grows at some moment i find myself stuck, i tried to increase all possiible limits on OSX, but meteor fails to start! I tried different packages, but after adding 5th or 6th meteor or npm package, it happens again! I completely remove node, nvm and reinstall meteor.

I made several new applications from scratch and everytime get the same result.

I even start to use only npm integrated into meteor: meteor npm install ...

How can i try other versions of meteor? Maybe downgrading to Meteor 1.3.0 solve the issue???

focused commented May 13, 2016

Please help! :(

When my project grows at some moment i find myself stuck, i tried to increase all possiible limits on OSX, but meteor fails to start! I tried different packages, but after adding 5th or 6th meteor or npm package, it happens again! I completely remove node, nvm and reinstall meteor.

I made several new applications from scratch and everytime get the same result.

I even start to use only npm integrated into meteor: meteor npm install ...

How can i try other versions of meteor? Maybe downgrading to Meteor 1.3.0 solve the issue???

@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 13, 2016

meteor run --release 1.3.1 does not help

focused commented May 13, 2016

meteor run --release 1.3.1 does not help

@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 13, 2016

Repeat with my friend all steps on freshly installed Ubuntu 14.04, everything works, so it's only OSX issue. It also works on Mac "13 OS El Captain 10.11.BETA, it only does not work on my Mac Retina "15...

focused commented May 13, 2016

Repeat with my friend all steps on freshly installed Ubuntu 14.04, everything works, so it's only OSX issue. It also works on Mac "13 OS El Captain 10.11.BETA, it only does not work on my Mac Retina "15...

@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 15, 2016

Finally solve the problem by creating new user in OS and reinstall meteor.

focused commented May 15, 2016

Finally solve the problem by creating new user in OS and reinstall meteor.

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 16, 2016

Member

I'd be surprised if this situation went away completely with a new user, unless your old user had a ~/.bashrc or ~/.bash_history line which was setting your max_files ulimit particularly low. (very possible though, as this setting used to be very common!). ulimit -n will show your setting for the current shell session.

Things will definitely work better on Ubuntu which has a much higher default maxfiles settings than OS X does, which is by default (I believe, set unreasonably low) at 256 files. It's possible a new user may have higher defaults these days, but I haven't created a new OS X user in ages, so I can't say for sure. It may happen that it will happen again as you add more files, but please feel free to try my 40,000 file reproduction repo and see if it works for you – it would be great if it did!

Meteor tries to make things better by setting your open-files ulimit as high as allowed but unfortunately, starting with OS X Yosemite, it became much more difficult to change ulimit "maximum" settings and it now involves changes to LaunchDaemon plists, etc. In fact, if you have it set manually to a number higher than 16384, Meteor will actually lower your maxfiles ulimit. (Though this part could easily be fixed).

Regardless, even if you could get Mac's ulimit -n higher, things will be slow with watching that many files. Performance could probably be improved if Meteor switched to using the more efficient fs.watch instead of fs.watchFile, which is what it uses right now. With the addition of watches on node_modules in 1.3 (and thus a much larger number of files), it might also be beneficial if Meteor tried using third-party file-watching solutions, perhaps the NPM packages chokidar or graceful-fs, FB's watchman or some combination thereof.

Member

abernix commented May 16, 2016

I'd be surprised if this situation went away completely with a new user, unless your old user had a ~/.bashrc or ~/.bash_history line which was setting your max_files ulimit particularly low. (very possible though, as this setting used to be very common!). ulimit -n will show your setting for the current shell session.

Things will definitely work better on Ubuntu which has a much higher default maxfiles settings than OS X does, which is by default (I believe, set unreasonably low) at 256 files. It's possible a new user may have higher defaults these days, but I haven't created a new OS X user in ages, so I can't say for sure. It may happen that it will happen again as you add more files, but please feel free to try my 40,000 file reproduction repo and see if it works for you – it would be great if it did!

Meteor tries to make things better by setting your open-files ulimit as high as allowed but unfortunately, starting with OS X Yosemite, it became much more difficult to change ulimit "maximum" settings and it now involves changes to LaunchDaemon plists, etc. In fact, if you have it set manually to a number higher than 16384, Meteor will actually lower your maxfiles ulimit. (Though this part could easily be fixed).

Regardless, even if you could get Mac's ulimit -n higher, things will be slow with watching that many files. Performance could probably be improved if Meteor switched to using the more efficient fs.watch instead of fs.watchFile, which is what it uses right now. With the addition of watches on node_modules in 1.3 (and thus a much larger number of files), it might also be beneficial if Meteor tried using third-party file-watching solutions, perhaps the NPM packages chokidar or graceful-fs, FB's watchman or some combination thereof.

@focused

This comment has been minimized.

Show comment
Hide comment
@focused

focused May 16, 2016

I tried your repo with new MacOS user:

owls:bunch-o-files owl$ meteor npm install
...
owls:bunch-o-files owl$ meteor -p 4000
[[[[[ ~/Development/meteor/bunch-o-files ]]]]]

=> Started proxy.
=> Started MongoDB.
=> Started your app.

=> App running at: http://localhost:4000/

/Users/owl/.meteor/packages/templating/.1.1.9.1h5n3gh++os+web.browser+web.cordova/plugin.compileTemplatesBatch.os/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:116
      throw error;
            ^
Error: UNKNOWN, open '/Users/owl/Development/meteor/bunch-o-files/public/32e/0/32e0493e40bb458a'
owls:bunch-o-files owl$

focused commented May 16, 2016

I tried your repo with new MacOS user:

owls:bunch-o-files owl$ meteor npm install
...
owls:bunch-o-files owl$ meteor -p 4000
[[[[[ ~/Development/meteor/bunch-o-files ]]]]]

=> Started proxy.
=> Started MongoDB.
=> Started your app.

=> App running at: http://localhost:4000/

/Users/owl/.meteor/packages/templating/.1.1.9.1h5n3gh++os+web.browser+web.cordova/plugin.compileTemplatesBatch.os/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:116
      throw error;
            ^
Error: UNKNOWN, open '/Users/owl/Development/meteor/bunch-o-files/public/32e/0/32e0493e40bb458a'
owls:bunch-o-files owl$
@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 19, 2016

Member

@focused Thanks for the results. Good to know.

Member

abernix commented May 19, 2016

@focused Thanks for the results. Good to know.

@pyry

This comment has been minimized.

Show comment
Hide comment
@pyry

pyry May 19, 2016

I had the same issue with Meteor 1.3.2.4 on OS X Yosemite 10.10.5 and running several instances at the same time (tests, react storybook and meteor itself):

find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5
122642 node_modules
 980 .meteor
  81 public
  60 imports
   4 client

however for me sudo launchctl limit maxfiles 524288 524288 seemed to solve the issue!

pyry commented May 19, 2016

I had the same issue with Meteor 1.3.2.4 on OS X Yosemite 10.10.5 and running several instances at the same time (tests, react storybook and meteor itself):

find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5
122642 node_modules
 980 .meteor
  81 public
  60 imports
   4 client

however for me sudo launchctl limit maxfiles 524288 524288 seemed to solve the issue!

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 19, 2016

Member

@pyry What does ulimit -n (without sudo) say for your system?

Member

abernix commented May 19, 2016

@pyry What does ulimit -n (without sudo) say for your system?

@pyry

This comment has been minimized.

Show comment
Hide comment
@pyry

pyry May 19, 2016

@abernix

ulimit -n
2560

pyry commented May 19, 2016

@abernix

ulimit -n
2560
@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix May 19, 2016

Member

Maybe node_modules and non-node_modules are watched differently then because if this fixed it for you, I find it strange because this code explicitly lowers your ulimit for the current process.

Only running sudo launchctl limit locally didn't work for me, but if anyone else is willing to try from a Meteor checkout, on OS X 10.10.x and 10.11.x I have personally only been able to improve this problem by:

  • Commenting out the Meteor code that changes the ulimit (same as linked above)
  • Raising my maxfiles up using this launchctl plist – though you should consult other instructions for better details. Make sure you reboot.
  • Make sure you do not have anything else setting your ulimit in your .bashrc or .bash_profile (or other shell config) as ulimit -n can only be set once per process (or only lowered thereafter).
  • If everything is correct, when you type ulimit -n it should print whatever value you have in that plist.

Meteor keeps spinning \ | / | \ | ... for a very long time after the app is initiated, but it seems to eventually stabilize (though I did still have a random seg-fault). Changes and reloads are still a bit slow, but they seem to work.

For now, I would still generally recommend avoiding having huge node dependency trees or keeping massive numbers of files around your project directory. Use Amazon S3 or similar to store excessive numbers of files where possible. Watching this many files (and thus providing the hot reloading that Meteor uses) will certainly be a bit sluggish. Once upgrades to Node are done, fs.watchFiles (and fs.watch) should improve a bit.

Member

abernix commented May 19, 2016

Maybe node_modules and non-node_modules are watched differently then because if this fixed it for you, I find it strange because this code explicitly lowers your ulimit for the current process.

Only running sudo launchctl limit locally didn't work for me, but if anyone else is willing to try from a Meteor checkout, on OS X 10.10.x and 10.11.x I have personally only been able to improve this problem by:

  • Commenting out the Meteor code that changes the ulimit (same as linked above)
  • Raising my maxfiles up using this launchctl plist – though you should consult other instructions for better details. Make sure you reboot.
  • Make sure you do not have anything else setting your ulimit in your .bashrc or .bash_profile (or other shell config) as ulimit -n can only be set once per process (or only lowered thereafter).
  • If everything is correct, when you type ulimit -n it should print whatever value you have in that plist.

Meteor keeps spinning \ | / | \ | ... for a very long time after the app is initiated, but it seems to eventually stabilize (though I did still have a random seg-fault). Changes and reloads are still a bit slow, but they seem to work.

For now, I would still generally recommend avoiding having huge node dependency trees or keeping massive numbers of files around your project directory. Use Amazon S3 or similar to store excessive numbers of files where possible. Watching this many files (and thus providing the hot reloading that Meteor uses) will certainly be a bit sluggish. Once upgrades to Node are done, fs.watchFiles (and fs.watch) should improve a bit.

@PEM--

This comment has been minimized.

Show comment
Hide comment
@PEM--

PEM-- May 31, 2016

Nasty bug. This settings did the trick for me: How to persist ulimit on OSX.

PEM-- commented May 31, 2016

Nasty bug. This settings did the trick for me: How to persist ulimit on OSX.

abernix added a commit to abernix/meteor that referenced this issue Sep 21, 2016

Do not lower the ulimit if it's already higher than 0x4000
Meteor tries to set the ulimit maxfiles as high as it can, however if it's already set to a value higher than 16384, it actually lowers it.  While this doesn't solve "Too many open files" for users with default OS settings, for those that have modified their system settings, this ensures that they are maintained.

Related to meteor/meteor#6952
@hwillson

This comment has been minimized.

Show comment
Hide comment
@hwillson

hwillson Nov 11, 2016

Member

ulimit -n 65536 65536 fails with too many arguments.

That's strange - that should definitely work?

screenshot 2016-11-11 10 48 53

After running ulimit -a:

screenshot 2016-11-11 10 49 44

Member

hwillson commented Nov 11, 2016

ulimit -n 65536 65536 fails with too many arguments.

That's strange - that should definitely work?

screenshot 2016-11-11 10 48 53

After running ulimit -a:

screenshot 2016-11-11 10 49 44

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Nov 11, 2016

Member

@hwillson I've definitely had issues with those commands in my days as well because of conflicts with other ulimit commands that had already been run. I believe the situation in this case is that it's already been set once.

@steve-ross Thanks for providing that. First, let me assure you that this is a complicated and unclear situation thanks to some OS X changes over the years. Is there any chance of you providing the output of the command I asked for as well? It's actually different than any of the things you showed. There are a lot of problems here actually:

  • First of all, the code snippet you pasted above is not from Meteor 1.4.2.x – not sure where you found it? The current launcher is actually much intelligent.
  • The way you are switching to root and changing ulimit and changing back to your normal user will never work. The ulimit can only be raised ONCE per shell session and when you change to root, you start a new shell, and when you leave it, you resume your usual shell (whatever it was before). Also, many times users have settings in their startup RC files which (try to) set the ulimit "high" but this is actually problematic as it can only be raised once per session and only lowered after that (even if it might have been possible originally!)
  • I'm really not sure /etc/sysctl.conf file works in any way on OS X El Capitan (or from Yosemite on). The /Library/LaunchDaemons/limit.maxfiles.plist is the option. Please read this page and remove anything that is no longer relevant on your version of OS X.
  • When you run launchctl limit, you can see that your OS is still setting a soft limit (one which you cannot go above as a regular user – ever) 5120 files.

Just for reference, here are my settings from my El Capitan machine:

$ cat /Library/LaunchDaemons/limit.maxproc.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxproc</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxproc</string>
          <string>2048</string>
          <string>2048</string>
        </array>
      <key>RunAtLoad</key>
        <true />
      <key>ServiceIPC</key>
        <false />
    </dict>
  </plist>
$ cat /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxfiles</string>
          <string>524288</string>
          <string>524288</string>
        </array>
      <key>RunAtLoad</key>
        <true/>
      <key>ServiceIPC</key>
        <false/>
    </dict>
  </plist>
$ sysctl -n kern.maxfilesperproc # this is the most important value
524288

Make sure you have nothing in your .zshrc or .bashrc that is setting your ulimit. I would also reboot after making changes to be sure. But the most important command on OS X El Capitan is the sysctl -n kern.maxfilesperproc output.

Member

abernix commented Nov 11, 2016

@hwillson I've definitely had issues with those commands in my days as well because of conflicts with other ulimit commands that had already been run. I believe the situation in this case is that it's already been set once.

@steve-ross Thanks for providing that. First, let me assure you that this is a complicated and unclear situation thanks to some OS X changes over the years. Is there any chance of you providing the output of the command I asked for as well? It's actually different than any of the things you showed. There are a lot of problems here actually:

  • First of all, the code snippet you pasted above is not from Meteor 1.4.2.x – not sure where you found it? The current launcher is actually much intelligent.
  • The way you are switching to root and changing ulimit and changing back to your normal user will never work. The ulimit can only be raised ONCE per shell session and when you change to root, you start a new shell, and when you leave it, you resume your usual shell (whatever it was before). Also, many times users have settings in their startup RC files which (try to) set the ulimit "high" but this is actually problematic as it can only be raised once per session and only lowered after that (even if it might have been possible originally!)
  • I'm really not sure /etc/sysctl.conf file works in any way on OS X El Capitan (or from Yosemite on). The /Library/LaunchDaemons/limit.maxfiles.plist is the option. Please read this page and remove anything that is no longer relevant on your version of OS X.
  • When you run launchctl limit, you can see that your OS is still setting a soft limit (one which you cannot go above as a regular user – ever) 5120 files.

Just for reference, here are my settings from my El Capitan machine:

$ cat /Library/LaunchDaemons/limit.maxproc.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxproc</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxproc</string>
          <string>2048</string>
          <string>2048</string>
        </array>
      <key>RunAtLoad</key>
        <true />
      <key>ServiceIPC</key>
        <false />
    </dict>
  </plist>
$ cat /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxfiles</string>
          <string>524288</string>
          <string>524288</string>
        </array>
      <key>RunAtLoad</key>
        <true/>
      <key>ServiceIPC</key>
        <false/>
    </dict>
  </plist>
$ sysctl -n kern.maxfilesperproc # this is the most important value
524288

Make sure you have nothing in your .zshrc or .bashrc that is setting your ulimit. I would also reboot after making changes to be sure. But the most important command on OS X El Capitan is the sysctl -n kern.maxfilesperproc output.

@steve-ross

This comment has been minimized.

Show comment
Hide comment
@steve-ross

steve-ross Nov 11, 2016

I appreciate that it's difficult like I said... I literally spent the whole day on this yesterday so I've done all this.

sysctl -n kern.maxfilesperproc                                                                                                        normal
65536

at /Library/LaunchDaemons/limit.maxfiles.plist                                                                                       normal
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxfiles</string>
          <string>65532</string>
          <string>65532</string>
        </array>
      <key>RunAtLoad</key>
        <true/>
      <key>ServiceIPC</key>
        <false/>
    </dict>
  </plist>

cat /Library/LaunchDaemons/limit.maxproc.plist                                                                                        normal
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxproc</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxproc</string>
          <string>2048</string>
          <string>2048</string>
        </array>
      <key>RunAtLoad</key>
        <true />
      <key>ServiceIPC</key>
        <false />
    </dict>
  </plist>

I appreciate that it's difficult like I said... I literally spent the whole day on this yesterday so I've done all this.

sysctl -n kern.maxfilesperproc                                                                                                        normal
65536

at /Library/LaunchDaemons/limit.maxfiles.plist                                                                                       normal
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxfiles</string>
          <string>65532</string>
          <string>65532</string>
        </array>
      <key>RunAtLoad</key>
        <true/>
      <key>ServiceIPC</key>
        <false/>
    </dict>
  </plist>

cat /Library/LaunchDaemons/limit.maxproc.plist                                                                                        normal
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxproc</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxproc</string>
          <string>2048</string>
          <string>2048</string>
        </array>
      <key>RunAtLoad</key>
        <true />
      <key>ServiceIPC</key>
        <false />
    </dict>
  </plist>

@steve-ross

This comment has been minimized.

Show comment
Hide comment
@steve-ross

steve-ross Nov 11, 2016

navigated to that command via the last commit message in this ticket regarding changing the ulimit... however it is still dev/nulled (see below) so it would be hard to tell if/when it doesn't work:

meteor/meteor

Line 125 in 340f0f6

ulimit -Sn ${maxfilesuse} > /dev/null 2>&1

navigated to that command via the last commit message in this ticket regarding changing the ulimit... however it is still dev/nulled (see below) so it would be hard to tell if/when it doesn't work:

meteor/meteor

Line 125 in 340f0f6

ulimit -Sn ${maxfilesuse} > /dev/null 2>&1

@steve-ross

This comment has been minimized.

Show comment
Hide comment
@steve-ross

steve-ross Nov 11, 2016

Just a little more background: I've even gone so far as to create a new user account and see if it was my ~/.bashrc or ~/.bash_proflie From speaking to another dev that I know he is also having issues on Sierra (he doesn't do meteor) and has spent quite a lot of time doing some of the same. Seems all Sierra has done is bumped the limit up a bit by default on the last release.

Just a little more background: I've even gone so far as to create a new user account and see if it was my ~/.bashrc or ~/.bash_proflie From speaking to another dev that I know he is also having issues on Sierra (he doesn't do meteor) and has spent quite a lot of time doing some of the same. Seems all Sierra has done is bumped the limit up a bit by default on the last release.

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Nov 11, 2016

Member

@steve-ross On your existing user, what happens if you open a completely brand new terminal shell and run?:

ulimit -Sn 65536

Also, I didn't include the output of mine earlier, but for the sake of completeness:

$ launchctl limit maxfiles
    maxfiles    524288         524288
$ sysctl -a | grep maxfiles
kern.maxfiles: 524288
kern.maxfilesperproc: 524288

Other things I want to point out, in your case, right now are:

  • The /etc/sysctl.conf values don't seem to matter anywhere, though I'm not surprised by this as it is not supposed to be used anymore.
  • I think you should try much higher values, for example 5120 is lower than your number of files in your project.
Member

abernix commented Nov 11, 2016

@steve-ross On your existing user, what happens if you open a completely brand new terminal shell and run?:

ulimit -Sn 65536

Also, I didn't include the output of mine earlier, but for the sake of completeness:

$ launchctl limit maxfiles
    maxfiles    524288         524288
$ sysctl -a | grep maxfiles
kern.maxfiles: 524288
kern.maxfilesperproc: 524288

Other things I want to point out, in your case, right now are:

  • The /etc/sysctl.conf values don't seem to matter anywhere, though I'm not surprised by this as it is not supposed to be used anymore.
  • I think you should try much higher values, for example 5120 is lower than your number of files in your project.
@steve-ross

This comment has been minimized.

Show comment
Hide comment
@steve-ross

steve-ross Nov 11, 2016

@abernix what point release are you on? Think this was a recent 'fix' by Apple

output: : (

♆ > ulimit -Sn 65536
-bash: ulimit: open files: cannot modify limit: Invalid argument
♆ > launchctl limit maxfiles
    maxfiles    65532          65532

@abernix what point release are you on? Think this was a recent 'fix' by Apple

output: : (

♆ > ulimit -Sn 65536
-bash: ulimit: open files: cannot modify limit: Invalid argument
♆ > launchctl limit maxfiles
    maxfiles    65532          65532
@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Nov 11, 2016

Member

@steve-ross: And what if you run?:

ulimit -Sn 256
Member

abernix commented Nov 11, 2016

@steve-ross: And what if you run?:

ulimit -Sn 256
@steve-ross

This comment has been minimized.

Show comment
Hide comment

@abernix That works...

@steve-ross

This comment has been minimized.

Show comment
Hide comment
@steve-ross

steve-ross Nov 11, 2016

@abernix so I was able to change it by doing the following but man this is a pain in the ass... makes me wonder if I have some leftover crap config files and I need a fresh install.

sudo -i
launchctl limit maxfiles 510000 510000
ulimit -Hn 510000
ulimit -Sn 510000
su sross

@abernix so I was able to change it by doing the following but man this is a pain in the ass... makes me wonder if I have some leftover crap config files and I need a fresh install.

sudo -i
launchctl limit maxfiles 510000 510000
ulimit -Hn 510000
ulimit -Sn 510000
su sross
@steve-ross

This comment has been minimized.

Show comment
Hide comment
@steve-ross

steve-ross Nov 11, 2016

HOLY SHIT.

♆ > cat /etc/profile
# System-wide .profile for sh(1)

if [ -x /usr/libexec/path_helper ]; then
    eval `/usr/libexec/path_helper -s`
fi

if [ "${BASH-no}" != "no" ]; then
    [ -r /etc/bashrc ] && . /etc/bashrc
fi
ulimit -n 4096

thx for all the help, the plist files are working now... gah!

HOLY SHIT.

♆ > cat /etc/profile
# System-wide .profile for sh(1)

if [ -x /usr/libexec/path_helper ]; then
    eval `/usr/libexec/path_helper -s`
fi

if [ "${BASH-no}" != "no" ]; then
    [ -r /etc/bashrc ] && . /etc/bashrc
fi
ulimit -n 4096

thx for all the help, the plist files are working now... gah!

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Nov 11, 2016

Member

Back at Facebook that's what we called a "clown shoe," meaning a bug that's really obvious when you stumble across it, but nobody can say quite how it got there.

Member

benjamn commented Nov 11, 2016

Back at Facebook that's what we called a "clown shoe," meaning a bug that's really obvious when you stumble across it, but nobody can say quite how it got there.

@steve-ross

This comment has been minimized.

Show comment
Hide comment

@benjamn stealing that

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Nov 15, 2016

Member

@steve-ross Haha. Glad to hear you got that sorted out!

If this turns out to be a recurring problem, some improvements to the meteor start-up script could be made to do some reconnaissance in search of mistakes that may be limiting the ability to raise the ulimit. For example, grep-ing various problem areas (/etc/profile, ~/{.bashrc,.zshrc,...}, etc.) upon ulimit failure. (Obviously this process should be named after some clown-themed horror movie.)

Member

abernix commented Nov 15, 2016

@steve-ross Haha. Glad to hear you got that sorted out!

If this turns out to be a recurring problem, some improvements to the meteor start-up script could be made to do some reconnaissance in search of mistakes that may be limiting the ability to raise the ulimit. For example, grep-ing various problem areas (/etc/profile, ~/{.bashrc,.zshrc,...}, etc.) upon ulimit failure. (Obviously this process should be named after some clown-themed horror movie.)

@veeramarni veeramarni referenced this issue in cdmbase/meter-react-redux-webpack-starter-kit Dec 21, 2016

Closed

increase ulimit on developer's machine #5

@kajisaap kajisaap referenced this issue in RocketChat/Rocket.Chat Dec 23, 2016

Closed

Pain points related to production usage so far #5321

@LDubya

This comment has been minimized.

Show comment
Hide comment
@LDubya

LDubya Jan 13, 2017

Having this issue on OS X Sierra. Can't solve it, have been trying for the past 2 days. It first came up when I ran Meteor Update to the latest version, 1.4.2.3. Previously it was at 1.3.4.3.

I checked bashrc, zshrc, etc, nothing there effecting this
I added the limit.maxfiles.plist (and restarted the computer), and I still get the error when I run meteor

As part of the build process, I package the app for electron using the meteor-electron from GitHub

I20170113-18:40:49.226(-5)? ATOM: [warn] kqueue: Too many open files
I20170113-18:40:49.226(-5)?
I20170113-18:40:49.227(-5)? ATOM: [warn] kqueue: Too many open files
I20170113-18:40:49.227(-5)? [err] evsignal_init: socketpair: Too many open files
I20170113-18:40:49.228(-5)? [err] evsignal_init: socketpair: Too many open files

LDubya commented Jan 13, 2017

Having this issue on OS X Sierra. Can't solve it, have been trying for the past 2 days. It first came up when I ran Meteor Update to the latest version, 1.4.2.3. Previously it was at 1.3.4.3.

I checked bashrc, zshrc, etc, nothing there effecting this
I added the limit.maxfiles.plist (and restarted the computer), and I still get the error when I run meteor

As part of the build process, I package the app for electron using the meteor-electron from GitHub

I20170113-18:40:49.226(-5)? ATOM: [warn] kqueue: Too many open files
I20170113-18:40:49.226(-5)?
I20170113-18:40:49.227(-5)? ATOM: [warn] kqueue: Too many open files
I20170113-18:40:49.227(-5)? [err] evsignal_init: socketpair: Too many open files
I20170113-18:40:49.228(-5)? [err] evsignal_init: socketpair: Too many open files
@LDubya

This comment has been minimized.

Show comment
Hide comment
@LDubya

LDubya Jan 24, 2017

Hey guys, I still am unable to fix this. I've tried everything I could think to try. I don't even have that many files. Everything was working fine until I decided to go through the trouble of running Meteor Update to the latest meteor version (and of course using the right Node version). Please help.

find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5

17342 private
8756 .meteor
3126 node_modules
 836 .git
 391 .meteor-electron
ulimit -n

524288

On OS X Sierra. After trying all reasonable limits, I set the max file size as high as possible to see if that would work and that's where I am now. No matter what number I set (and restart the computer) it doesn't stop the console from erring out too many open files. It is set inside the limit.maxfiles.plist file. It's not set anywhere else (not in the profile, bashrc, zshrc, etc).

Please see my previous comment for the error logs I get.

Someone please help =(

LDubya commented Jan 24, 2017

Hey guys, I still am unable to fix this. I've tried everything I could think to try. I don't even have that many files. Everything was working fine until I decided to go through the trouble of running Meteor Update to the latest meteor version (and of course using the right Node version). Please help.

find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5

17342 private
8756 .meteor
3126 node_modules
 836 .git
 391 .meteor-electron
ulimit -n

524288

On OS X Sierra. After trying all reasonable limits, I set the max file size as high as possible to see if that would work and that's where I am now. No matter what number I set (and restart the computer) it doesn't stop the console from erring out too many open files. It is set inside the limit.maxfiles.plist file. It's not set anywhere else (not in the profile, bashrc, zshrc, etc).

Please see my previous comment for the error logs I get.

Someone please help =(

@LDubya

This comment has been minimized.

Show comment
Hide comment
@LDubya

LDubya Feb 5, 2017

Bump. Help? Anyone?

LDubya commented Feb 5, 2017

Bump. Help? Anyone?

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Feb 9, 2017

Member

@LDubya Does the problem go away if you get rid of the 17342 files in your private/ folder?

Member

abernix commented Feb 9, 2017

@LDubya Does the problem go away if you get rid of the 17342 files in your private/ folder?

@LDubya

This comment has been minimized.

Show comment
Hide comment
@LDubya

LDubya Feb 17, 2017

@abernix I still get too many open files.

I20170217-13:18:49.882(-5)? ATOM: [warn] kqueue: Too many open files
I20170217-13:18:49.882(-5)? [warn] kqueue: Too many open files
I20170217-13:18:49.883(-5)? [err] evsignal_init: socketpair: Too many open files

and if I list my file count...

$ find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5

8969 .meteor
2778 node_modules
 207 .meteor-electron
 112 public
  37 client

Someone suggested that maybe there are other apps using a lot of files. But listing the top ten number of active file descriptors doesn't show anything weird.

$ lsof | awk '{print $1}' | uniq -c | sort -rn | head

 432 Dock
 388 Spotlight
 350 Finder
 260 Safari
 219 AppleSpel
 125 nsurlstor
 121 UserEvent
 117 bird
 116 iTerm2
 106 Notificat

any ideas? I have never been able to get this to work and it's wearing me out.

I have also closed every app besides Safari (with 2 tabs open), Terminal, and Finder.

LDubya commented Feb 17, 2017

@abernix I still get too many open files.

I20170217-13:18:49.882(-5)? ATOM: [warn] kqueue: Too many open files
I20170217-13:18:49.882(-5)? [warn] kqueue: Too many open files
I20170217-13:18:49.883(-5)? [err] evsignal_init: socketpair: Too many open files

and if I list my file count...

$ find . -type f | cut -d / -f 2 | sort | uniq -c | sort -rn | head -5

8969 .meteor
2778 node_modules
 207 .meteor-electron
 112 public
  37 client

Someone suggested that maybe there are other apps using a lot of files. But listing the top ten number of active file descriptors doesn't show anything weird.

$ lsof | awk '{print $1}' | uniq -c | sort -rn | head

 432 Dock
 388 Spotlight
 350 Finder
 260 Safari
 219 AppleSpel
 125 nsurlstor
 121 UserEvent
 117 bird
 116 iTerm2
 106 Notificat

any ideas? I have never been able to get this to work and it's wearing me out.

I have also closed every app besides Safari (with 2 tabs open), Terminal, and Finder.

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Feb 17, 2017

Member

@LDubya You can disable one of the primary sources of open files by forcing the file watcher to use polling instead of OS file change notifications:

export METEOR_WATCH_FORCE_POLLING=1
meteor run

Note that this will cause the file watcher to stat your files every 500ms, which may increase idle CPU usage, but that is probably less of a problem than EMFILE errors.

If 500ms is too frequent (or not frequent enough!), you can set the polling interval explicitly:

export METEOR_WATCH_POLLING_INTERVAL_MS=1000
Member

benjamn commented Feb 17, 2017

@LDubya You can disable one of the primary sources of open files by forcing the file watcher to use polling instead of OS file change notifications:

export METEOR_WATCH_FORCE_POLLING=1
meteor run

Note that this will cause the file watcher to stat your files every 500ms, which may increase idle CPU usage, but that is probably less of a problem than EMFILE errors.

If 500ms is too frequent (or not frequent enough!), you can set the polling interval explicitly:

export METEOR_WATCH_POLLING_INTERVAL_MS=1000
@LDubya

This comment has been minimized.

Show comment
Hide comment
@LDubya

LDubya Feb 17, 2017

@benjamn this seems to be allowing meteor to run, finally. Thanks!

If I wanted to return it back to the default, is it as easy as:?

export METEOR_WATCH_FORCE_POLLING=0

LDubya commented Feb 17, 2017

@benjamn this seems to be allowing meteor to run, finally. Thanks!

If I wanted to return it back to the default, is it as easy as:?

export METEOR_WATCH_FORCE_POLLING=0
@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Feb 17, 2017

Member

That's right, or unset METEOR_WATCH_FORCE_POLLING. Note that environment variables only apply to the current shell/terminal window, so you might need to put the export in a .bashrc file (depending on your shell) if you want it to be set for all new shells.

Member

benjamn commented Feb 17, 2017

That's right, or unset METEOR_WATCH_FORCE_POLLING. Note that environment variables only apply to the current shell/terminal window, so you might need to put the export in a .bashrc file (depending on your shell) if you want it to be set for all new shells.

@lpgeiger lpgeiger referenced this issue in meteortesting/meteor-mocha Feb 22, 2017

Closed

Meteor Todo App tests hanging with nightmare #11

@lpgeiger

This comment has been minimized.

Show comment
Hide comment
@lpgeiger

lpgeiger Feb 22, 2017

+1 export METEOR_WATCH_FORCE_POLLING=1 solved it for me. @benjamn are you recommending this be the default setting for OSX?

+1 export METEOR_WATCH_FORCE_POLLING=1 solved it for me. @benjamn are you recommending this be the default setting for OSX?

@partounian

This comment has been minimized.

Show comment
Hide comment
@partounian

partounian Mar 12, 2017

@LDubya That link will show you how to create plist files that make the file limit values stick, I used different numbers but the point is still useful of file format and location.
https://www.chrissearle.org/2016/10/01/too-many-open-files-on-osx-macos/

@LDubya That link will show you how to create plist files that make the file limit values stick, I used different numbers but the point is still useful of file format and location.
https://www.chrissearle.org/2016/10/01/too-many-open-files-on-osx-macos/

@matmar10

This comment has been minimized.

Show comment
Hide comment
@matmar10

matmar10 May 9, 2017

+1 export METEOR_WATCH_FORCE_POLLING=1 also solved it for me. Thanks @benjamn

matmar10 commented May 9, 2017

+1 export METEOR_WATCH_FORCE_POLLING=1 also solved it for me. Thanks @benjamn

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Jul 17, 2017

Member

We believe this problem should be substantially improved by changes in Meteor 1.5.1 which drastically reduce the number of "watched" files and thus the number of concurrently open file descriptors. While this change also means that you may have to wait up to 5 seconds for the first change to be detected on a particular file, further changes should be quite quick once the watcher is created. To quote the History.md from Meteor 1.5.1:

open file descriptors will now be bounded roughly by the number of files you are actively editing, rather than the number of files involved in the build (often thousands)

See also: #8866.

Member

abernix commented Jul 17, 2017

We believe this problem should be substantially improved by changes in Meteor 1.5.1 which drastically reduce the number of "watched" files and thus the number of concurrently open file descriptors. While this change also means that you may have to wait up to 5 seconds for the first change to be detected on a particular file, further changes should be quite quick once the watcher is created. To quote the History.md from Meteor 1.5.1:

open file descriptors will now be bounded roughly by the number of files you are actively editing, rather than the number of files involved in the build (often thousands)

See also: #8866.

@abernix abernix closed this Jul 17, 2017

@molerat619

This comment has been minimized.

Show comment
Hide comment
@molerat619

molerat619 Aug 22, 2017

I've run these commands:

echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
ulimit -n 65536

And since then my mysql database in my other nodejs application stopped working. I opened this issue: sequelize/sequelize#8120 Do you think it can have anything to do with it?

molerat619 commented Aug 22, 2017

I've run these commands:

echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
ulimit -n 65536

And since then my mysql database in my other nodejs application stopped working. I opened this issue: sequelize/sequelize#8120 Do you think it can have anything to do with it?

@benjamn

This comment has been minimized.

Show comment
Hide comment
@benjamn

benjamn Aug 22, 2017

Member

@molerat619 What version of Meteor are you using? Meteor 1.5.1 contains substantial improvements for this issue.

Member

benjamn commented Aug 22, 2017

@molerat619 What version of Meteor are you using? Meteor 1.5.1 contains substantial improvements for this issue.

@molerat619

This comment has been minimized.

Show comment
Hide comment
@molerat619

molerat619 Aug 22, 2017

@benjamn I'm not using Meteor at all but I've run the same commands and they solved my other issue I had back then. So I am asking now if that could have caused my new problem since it also has to do with sizes and nothing else makes sense to me.

@benjamn I'm not using Meteor at all but I've run the same commands and they solved my other issue I had back then. So I am asking now if that could have caused my new problem since it also has to do with sizes and nothing else makes sense to me.

@molerat619

This comment has been minimized.

Show comment
Hide comment
@molerat619

molerat619 Aug 22, 2017

Never mind, I have found the problem and it was not caused by these commands. Sorry

Never mind, I have found the problem and it was not caused by these commands. Sorry

@meteor meteor locked and limited conversation to collaborators Aug 23, 2017

@abernix

This comment has been minimized.

Show comment
Hide comment
@abernix

abernix Aug 23, 2017

Member

Any new Meteor users experiencing this same problem should open a new issue. Thanks!

Member

abernix commented Aug 23, 2017

Any new Meteor users experiencing this same problem should open a new issue. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.