EMFILE installing a package #3259

Closed
juliangruber opened this Issue Mar 17, 2013 · 56 comments

Projects

None yet
Contributor
$ node -v
v0.10.0
$ npm -v
1.2.14
$ sudo npm install -g component
npm http GET https://registry.npmjs.org/component
npm http 200 https://registry.npmjs.org/component
npm http GET https://registry.npmjs.org/component/-/component-0.13.0.tgz
npm http 200 https://registry.npmjs.org/component/-/component-0.13.0.tgz
npm http GET https://registry.npmjs.org/superagent/0.10.0
npm http GET https://registry.npmjs.org/component-builder/0.6.3
npm http GET https://registry.npmjs.org/string-to-js/0.0.1
npm http GET https://registry.npmjs.org/jog/0.4.0
npm http GET https://registry.npmjs.org/batch/0.2.1
npm http GET https://registry.npmjs.org/win-fork/1.0.0
npm http GET https://registry.npmjs.org/archy/0.0.2
npm http GET https://registry.npmjs.org/debug
npm http GET https://registry.npmjs.org/commander/1.0.5
npm http GET https://registry.npmjs.org/mkdirp/0.3.4
npm http 200 https://registry.npmjs.org/string-to-js/0.0.1
npm http GET https://registry.npmjs.org/string-to-js/-/string-to-js-0.0.1.tgz
npm http 200 https://registry.npmjs.org/component-builder/0.6.3
npm http GET https://registry.npmjs.org/component-builder/-/component-builder-0.6.3.tgz
npm http 200 https://registry.npmjs.org/jog/0.4.0
npm http GET https://registry.npmjs.org/jog/-/jog-0.4.0.tgz
npm http 200 https://registry.npmjs.org/win-fork/1.0.0
npm http GET https://registry.npmjs.org/win-fork/-/win-fork-1.0.0.tgz
npm http 200 https://registry.npmjs.org/superagent/0.10.0
npm http GET https://registry.npmjs.org/superagent/-/superagent-0.10.0.tgz
npm http 200 https://registry.npmjs.org/batch/0.2.1
npm http GET https://registry.npmjs.org/batch/-/batch-0.2.1.tgz
npm http 200 https://registry.npmjs.org/archy/0.0.2
npm http GET https://registry.npmjs.org/archy/-/archy-0.0.2.tgz
npm http 200 https://registry.npmjs.org/mkdirp/0.3.4
npm http 200 https://registry.npmjs.org/string-to-js/-/string-to-js-0.0.1.tgz
npm http GET https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.4.tgz
npm http 200 https://registry.npmjs.org/component-builder/-/component-builder-0.6.3.tgz
npm http 200 https://registry.npmjs.org/debug
npm http GET https://registry.npmjs.org/debug/-/debug-0.7.2.tgz
npm http 200 https://registry.npmjs.org/commander/1.0.5
npm http GET https://registry.npmjs.org/commander/-/commander-1.0.5.tgz
npm http 200 https://registry.npmjs.org/jog/-/jog-0.4.0.tgz
npm http 200 https://registry.npmjs.org/win-fork/-/win-fork-1.0.0.tgz
npm http 200 https://registry.npmjs.org/superagent/-/superagent-0.10.0.tgz
npm http 200 https://registry.npmjs.org/batch/-/batch-0.2.1.tgz
npm http 200 https://registry.npmjs.org/archy/-/archy-0.0.2.tgz
npm http 200 https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.4.tgz
npm http 200 https://registry.npmjs.org/debug/-/debug-0.7.2.tgz
npm http 200 https://registry.npmjs.org/commander/-/commander-1.0.5.tgz
npm http GET https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/component-require/0.2.0
npm http GET https://registry.npmjs.org/redis/0.7.1
npm http GET https://registry.npmjs.org/ms/0.1.0
npm http GET https://registry.npmjs.org/better-assert
npm http GET https://registry.npmjs.org/commander/0.5.2
npm http GET https://registry.npmjs.org/keypress
npm http GET https://registry.npmjs.org/qs/0.5.2
npm http GET https://registry.npmjs.org/formidable/1.0.9
npm http GET https://registry.npmjs.org/mime/1.2.5
npm http GET https://registry.npmjs.org/emitter-component/0.0.6
npm http GET https://registry.npmjs.org/methods/0.0.1
npm http GET https://registry.npmjs.org/cookiejar/1.3.0
npm http 200 https://registry.npmjs.org/ms/0.1.0
npm http GET https://registry.npmjs.org/ms/-/ms-0.1.0.tgz
npm http 200 https://registry.npmjs.org/redis/0.7.1
npm http GET https://registry.npmjs.org/redis/-/redis-0.7.1.tgz
npm http 200 https://registry.npmjs.org/better-assert
npm http GET https://registry.npmjs.org/better-assert/-/better-assert-0.1.0.tgz
npm http 200 https://registry.npmjs.org/component-require/0.2.0
npm http GET https://registry.npmjs.org/component-require/-/component-require-0.2.0.tgz
npm http 200 https://registry.npmjs.org/commander/0.5.2
npm http GET https://registry.npmjs.org/commander/-/commander-0.5.2.tgz
npm http 200 https://registry.npmjs.org/keypress
npm http GET https://registry.npmjs.org/keypress/-/keypress-0.1.0.tgz
npm http 200 https://registry.npmjs.org/qs/0.5.2
npm http GET https://registry.npmjs.org/qs/-/qs-0.5.2.tgz
npm http 200 https://registry.npmjs.org/mime/1.2.5
npm http GET https://registry.npmjs.org/mime/-/mime-1.2.5.tgz
npm http 200 https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/fs-extra/-/fs-extra-0.5.0.tgz
npm http 200 https://registry.npmjs.org/formidable/1.0.9
npm http GET https://registry.npmjs.org/formidable/-/formidable-1.0.9.tgz
npm http 200 https://registry.npmjs.org/methods/0.0.1
npm http GET https://registry.npmjs.org/methods/-/methods-0.0.1.tgz
npm http 200 https://registry.npmjs.org/cookiejar/1.3.0
npm http GET https://registry.npmjs.org/cookiejar/-/cookiejar-1.3.0.tgz
npm http 200 https://registry.npmjs.org/emitter-component/0.0.6
npm http 200 https://registry.npmjs.org/ms/-/ms-0.1.0.tgz
npm http GET https://registry.npmjs.org/emitter-component/-/emitter-component-0.0.6.tgz
npm http 200 https://registry.npmjs.org/redis/-/redis-0.7.1.tgz
npm http 200 https://registry.npmjs.org/better-assert/-/better-assert-0.1.0.tgz
npm http 200 https://registry.npmjs.org/component-require/-/component-require-0.2.0.tgz
npm http 200 https://registry.npmjs.org/commander/-/commander-0.5.2.tgz
npm http 200 https://registry.npmjs.org/keypress/-/keypress-0.1.0.tgz
npm http 200 https://registry.npmjs.org/qs/-/qs-0.5.2.tgz
npm http 200 https://registry.npmjs.org/mime/-/mime-1.2.5.tgz
npm http 200 https://registry.npmjs.org/fs-extra/-/fs-extra-0.5.0.tgz
npm http 200 https://registry.npmjs.org/formidable/-/formidable-1.0.9.tgz
npm http 200 https://registry.npmjs.org/methods/-/methods-0.0.1.tgz
npm WARN package.json methods@0.0.1 No README.md file found!
npm http GET https://registry.npmjs.org/callsite
npm http GET https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/jsonfile
npm http 200 https://registry.npmjs.org/cookiejar/-/cookiejar-1.3.0.tgz
npm http GET https://registry.npmjs.org/rimraf
npm http 200 https://registry.npmjs.org/emitter-component/-/emitter-component-0.0.6.tgz
npm http 200 https://registry.npmjs.org/jsonfile
npm http GET https://registry.npmjs.org/jsonfile/-/jsonfile-0.0.1.tgz
npm http 200 https://registry.npmjs.org/rimraf
npm http GET https://registry.npmjs.org/rimraf/-/rimraf-2.1.4.tgz
npm http 200 https://registry.npmjs.org/callsite
npm http GET https://registry.npmjs.org/callsite/-/callsite-1.0.0.tgz
npm http 200 https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/ncp/-/ncp-0.2.7.tgz
npm http 200 https://registry.npmjs.org/jsonfile/-/jsonfile-0.0.1.tgz
npm http 200 https://registry.npmjs.org/rimraf/-/rimraf-2.1.4.tgz
npm http 200 https://registry.npmjs.org/callsite/-/callsite-1.0.0.tgz
npm http 200 https://registry.npmjs.org/ncp/-/ncp-0.2.7.tgz
npm http GET https://registry.npmjs.org/graceful-fs
npm http 200 https://registry.npmjs.org/graceful-fs
npm http GET https://registry.npmjs.org/graceful-fs/-/graceful-fs-1.2.0.tgz
npm http 200 https://registry.npmjs.org/graceful-fs/-/graceful-fs-1.2.0.tgz
/usr/local/bin/component-info -> /usr/local/lib/node_modules/component/bin/component-info
/usr/local/bin/component -> /usr/local/lib/node_modules/component/bin/component
/usr/local/bin/component-ls -> /usr/local/lib/node_modules/component/bin/component-ls
/usr/local/bin/component-build -> /usr/local/lib/node_modules/component/bin/component-build
/usr/local/bin/component-convert -> /usr/local/lib/node_modules/component/bin/component-convert
/usr/local/bin/component-install -> /usr/local/lib/node_modules/component/bin/component-install
/usr/local/bin/component-help -> /usr/local/lib/node_modules/component/bin/component-help
/usr/local/bin/component-search -> /usr/local/lib/node_modules/component/bin/component-search
/usr/local/bin/component-create -> /usr/local/lib/node_modules/component/bin/component-create
/usr/local/bin/component-changes -> /usr/local/lib/node_modules/component/bin/component-changes
/usr/local/bin/component-docs -> /usr/local/lib/node_modules/component/bin/component-docs
/usr/local/bin/component-wiki -> /usr/local/lib/node_modules/component/bin/component-wiki
npm ERR! Error: EMFILE, too many open files
npm ERR!     at new Glob (/usr/local/lib/node_modules/npm/node_modules/glob/glob.js:104:21)
npm ERR!     at glob (/usr/local/lib/node_modules/npm/node_modules/glob/glob.js:57:11)
npm ERR!     at gypfile (/usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:150:17)
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:137:40
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:54:35
npm ERR!     at Array.forEach (native)
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:54:11
npm ERR!     at Array.forEach (native)
npm ERR!     at asyncMap (/usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:53:8)
npm ERR!     at extras (/usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:136:17)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

npm ERR! System Darwin 12.2.1
npm ERR! command "node" "/usr/local/bin/npm" "i" "-g" "component"

/usr/local/lib/node_modules/npm/lib/utils/error-handler.js:247
  log.error("cwd", process.cwd())
                           ^
Error: EMFILE, too many open files
    at process.errorHandler (/usr/local/lib/node_modules/npm/lib/utils/error-handler.js:247:28)
    at process.EventEmitter.emit (events.js:117:20)
    at process._fatalException (node.js:272:26)

I got this too on a installing a repo with a lot of dependencies:

npm ERR! fetch failed https://registry.npmjs.org/connect/-/connect-2.4.6.tgz
npm ERR! Error: EMFILE, open '/Users/faridneshat/.npm/f9af8e28-lob-whatev-glob-whatev-0-1-8-tgz.lock'
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

npm ERR! System Darwin 10.8.0
npm ERR! command "/Users/faridneshat/nvm/v0.10.1/bin/node" "/Users/faridneshat/nvm/v0.10.1/bin/npm" "install"
npm ERR! Error: EMFILE, too many open files
npm ERR!     at errorHandler (/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/utils/error-handler.js:247:28)
npm ERR!     at cb (/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/install.js:79:20)
npm ERR!     at /Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/install.js:153:26
npm ERR!     at cb (/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/install.js:475:40)
npm ERR!     at cb (/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
npm ERR!     at /Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/install.js:706:5
npm ERR!     at /Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/install.js:805:24
npm ERR!     at /Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/node_modules/slide/lib/chain.js:16:22
npm ERR!     at /Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/install.js:967:14
npm ERR!     at cb (/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

npm ERR! System Darwin 10.8.0
npm ERR! command "/Users/faridneshat/nvm/v0.10.1/bin/node" "/Users/faridneshat/nvm/v0.10.1/bin/npm" "install"

/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/utils/error-handler.js:247
  log.error("cwd", process.cwd())
                           ^
Error: EMFILE, too many open files
    at process.errorHandler (/Users/faridneshat/nvm/v0.10.1/lib/node_modules/npm/lib/utils/error-handler.js:247:28)
    at process.EventEmitter.emit (events.js:117:20)
    at process._fatalException (node.js:272:26)

The limit(from ulimit) was 256. Increasing it to 2048 fixed it.

Contributor
edef1c commented Mar 28, 2013

Should we be handling this in npm? Or should we consider too low a ulimit user error?

Contributor

npm shouldn't require a custom ulimit. That's what @isaacs made graceful-fs for.

Contributor
edef1c commented Mar 28, 2013

Okay, brilliant. Pullreq welcome.

@edef1c edef1c was assigned Mar 28, 2013
Contributor

it's used already :D that's the weird thing.

I'm seeing this one a ton too, all of a sudden. Never saw it until recently. ulimit works around the problem.

I'm getting this today with unlimited ulimit...

Owner
isaacs commented Apr 10, 2013

@JoshuaGross What does ulimit -a report?

Owner
isaacs commented Apr 10, 2013

This is a bug in graceful-fs. It's catching EMFILEs from fs.open, but not from fs.readdir, which also uses a file descriptor. Patch coming soon.

Aside, I'm not sure if I'd call it a bug in npm, per se, but it's certainly reading a lot more files and dirs than it really ought to have to in order to do an ls, so that's not very efficient. Probably a ** in a glob somewhere making it traverse the entire tree.

In the meantime, you can do sudo ulimit -n 2560 (or "unlimited" or some other big number) which seems to make this go away in the short term.

Not really sure what was going on last night, I did ulimit -a and it
reported "unlimited". So, obviously that made no sense. I restarted my
computer at that point, and then I ran into a bunch of peer dependency
issues after that (but not EMFILE). tl;dr probably a temporary issue on my
end?

Joshua Gross
Christian / SpanDeX, Inc. / BA of Computer Science, UW-Madison 2013
http://www.joshisgross.com / http://www.spandex.io

On 10 April 2013 12:03, Isaac Z. Schlueter notifications@github.com wrote:

This is a bug in graceful-fs. It's catching EMFILEs from fs.open, but not
from fs.readdir, which also uses a file descriptor. Patch coming soon.

Aside, I'm not sure if I'd call it a bug in npm, per se, but it's
certainly reading a lot more files and dirs than it really ought to have to
in order to do an ls, so that's not very efficient. Probably a ** in a
glob somewhere making it traverse the entire tree.

In the meantime, you can do sudo ulimit -n 2560 (or "unlimited" or some
other big number) which seems to make this go away in the short term.


Reply to this email directly or view it on GitHubhttps://github.com/isaacs/npm/issues/3259#issuecomment-16187575
.

I'm getting this too, trying to install bower, component, grunt-cli, etc. globally.

Everything seems to be resolved after I reinstalled node 0.10.3

reproducible with node 0.10.5 on OS X. node was installed from MacPorts if that matters

nicknisi commented May 3, 2013

I'm still having the issue. I have 0.10.3 installed on OSX 10.8 with nvm, but the nvm part doesn't seem to matter. I get this every time I install something globally. I get around it by running nvm use 0.10.3, which seems to allow me to install the next thing globally.

@isaacs isaacs was assigned May 5, 2013
base698 commented May 7, 2013

Still have the issue in 0.10.5 Mac default ulimit only 256

jcw commented May 10, 2013

Got this error here too, npm 1.2.18, node 0.10.5, macosx 10.8.3, ulimit default 256.

Edit: the command was "npm ls -g" - but "npm update -g" works fine.

+1

Just to chime in, this is still an issue. npm install -g grunt-cli throws numerous errors about missing keys and EMFILE, even navigating to a local directory and running npm install throws the EMFILE error.

atufkas commented May 27, 2013

+1 Same with me. OSX 10.8, node 0.10.7 (via Installer), npm 1.2.23. Issue "EMFILE, too many open files" now occurs in global npm update. Latest installs were yo, bower and grunt-cli. I know that's probably several issues but: Overall npm install/update errors and warnings (unmet dependencies, that "no repository field" thing) heavily increased since upgrading from node 0.8.x and installing latest npm with that.

Stop posting "me too" comments. OSX's 256 max-filedescriptors in default are ought to be erroneous. Try sudo ulimit -n 4096 & ulimit -n 4096 (userland) and you'll get temporarily around that. By the way, running npm install et al. using sudo will often get you even further to the goal.

medikoo commented May 30, 2013

@KenanSulayman you shouldn't hack your system to have NPM working, it should be resolved internally within NPM, that's possible, there are known solutions for that.

Owner
isaacs commented May 30, 2013

In the latest npm release, this is fixed. Node v0.10.9 will be released today with that npm update. This issue is closed. In the meantime, if you can't upgrade, bump up your unlimit -n.

@isaacs isaacs closed this May 30, 2013
Owner
isaacs commented May 30, 2013

Note: Setting the ulimit -n 16 does make it die, since it ends up not having an available file descriptor to do the DNS lookup for the registry. But with ulimit -n 64, it works fine.

+1 to @medikoo. Unnecessary use of sudo is bad news.

Thanks for the fix @isaacs.

hudgins commented May 31, 2013

@isaacs This appears to still be happening (with v0.10.9 and npm 1.2.24) if you try to install a module with --dev (try 'npm install --dev tap').

Still happening to me on the latest install when installing globally.

node -v
v0.10.9
npm -v
1.2.24
Brainux commented May 31, 2013

ulimit -n 16, 64, 128, 256, 512, 1024, 2048, I stopped here.

Still happening to me on the latest install when installing globally.

node -v
v0.10.9
npm -v
1.2.24

Brainux commented Jun 1, 2013

After restarting it is ok,like @JoshuaGross said above.

Owner
isaacs commented Jun 1, 2013

@Brainux Restarting what?

What are you guys installing that's causing this error? Is there any way I can reproduce it?

hudgins commented Jun 1, 2013

@isaacs Did you try 'npm install --dev tap'? I tried rebooting my Mac and that does seem to have improved things for some reason, but it still dies eventually with errors like:

npm ERR! Error: ENOENT, utime '/Users/ahudgins/bug/node_modules/tap/node_modules/glob/node_modules/minimatch/node_modules/lru-cache/node_modules/weak/node_modules/mocha/node_modules/coffee-script/node_modules/jison/node_modules/browserify/node_modules/browser-resolve/node_modules/crypto-browserify/node_modules/tape/.travis.yml'

and ultimately:

npm ERR! Error: EMFILE, open '/Users/ahudgins/bug/node_modules/tap/node_modules/glob/node_modules/minimatch/node_modules/lru-cache/node_modules/weak/node_modules/mocha/node_modules/coffee-script/node_modules/jison/node_modules/browserify/node_modules/umd/node_modules/rfile/node_modules/callsite/Makefile'

I try ulimit command with 10000+ and not solve my problem, neither rebooting. I have a Mac.

Brainux commented Jun 2, 2013

@isaacs, in my case, restart == reboot. I´m using:
Ubuntu
node -v
v0.10.9
npm -v
1.2.24

My partial .bash_history
...
npm install -g buster phantom
npm install -g phantom
npm install -g buster
apt-get update
apt-get install nodejs //nothing new
npm install -g buster
npm install -verbose -g buster
npm install -g phantom
npm uninstall -g phantom
npm install -g phantom
ulimit -n 64
npm install -g phantom
npm install -verbose -g phantom
npm update -g
ulimit -n 128
npm update -g
ulimit -n 256
npm update -g
ulimit -n 512
npm update -g
ulimit -n 1024
npm update -g
ulimit -n 2048
npm update -g
unlimit -a
ulimit -a
ulimit -n 2048
ulimit -a
npm install -g phantom
npm install -g generator-karma@~0.2.0
npm install -g
npm install -g buster
// ahahahahhhhhhhhhhhh!!!!
init 6
npm install -g phantom
npm install -g buster
:)

// one day after:
npm install -g wordpress.js //error again
npm install -g wordpress.js //error again
npm install -g -verbose wordpress.js //error again
npm install -g -verbose wordpress.js //error again
npm install -g wordpress.js // error again
:(
npm install -g wordpress.js // it´s working!!!! No "init 6" !!!
//very annoying issue
//To test:
npm uninstall -g -verbose phantom //ok
npm install -g phantom //ok
npm uninstall -g -verbose buster //ok
npm install -g buster //ok
npm uninstall -g wordpress.js //ok
npm install -g wordpress.js //ok

after some erros, node just worked by itself.

Is there a module to generate a standard report and post here?

jcw commented Jun 6, 2013

Node 0.10.10, npm 1.2.25, OSX 10.8.4, still getting the EMFILE with "npm install -g hubot" (npm update -g worked).

Note ulimit -n is at default 256 - I don't think the answer should be "adjust your OS settings to make npm work", the ball really is in npm's court, I'm afraid - it's the only command generating this type of error.

I don't understand the I don't think the answer should be "adjust your OS settings to make npm work" issue — no operating system except OSX has a default fd-max of 0x100. Why should every wheel be reinvented, just because Apple thinks that's right?

Member
domenic commented Jun 6, 2013

@isaacs You said earlier:

This is a bug in graceful-fs. It's catching EMFILEs from fs.open, but not from fs.readdir, which also uses a file descriptor. Patch coming soon.

I didn't see a patch to graceful-fs. Perhaps that could explain what our users are seeing?

jcw commented Jun 6, 2013

@KenanSulayman - npm is a tool (one of many), OSX is the environment in which it runs. I see two reasons:

  1. I don't really want to let a tool dictate the global settings of my system (where would one draw the line?)
  2. solving it once within npm means that lots of people don't have to address (or even know about) this issue

Having said that, I agree that the best solution would be for Apple to make better default choices.

With apologies if my choice of words came across as too harsh.

I'm having issue with updating globally installed packge in npm:

$ node -v
v0.10.10

$ npm -v
1.2.30

$ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: OS X 10.8.4 (12E55)
      Kernel Version: Darwin 12.4.0
$ sudo npm update -g generator-server-configs
npm WARN package.json bytes@0.2.0 No repository field.
npm WARN package.json cookie-signature@1.0.1 No repository field.
npm WARN package.json fresh@0.1.0 No repository field.
npm WARN package.json pause@0.0.1 No repository field.
npm WARN package.json dateformat@1.0.2-1.2.3 No repository field.
npm WARN package.json eventemitter2@0.4.11 No repository field.
npm WARN package.json eventemitter2@0.4.11 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
npm WARN package.json cli-table@0.2.0 No repository field.
npm ERR! Error: EMFILE, open '/usr/local/lib/node_modules/yeoman-generator/node_modules/request/node_modules/forever-agent/package.json'
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

npm ERR! System Darwin 12.4.0
npm ERR! command "node" "/usr/local/bin/npm" "update" "-g" "generator-server-configs"
npm ERR! Error: EMFILE, too many open files
npm ERR!     at errorHandler (/usr/local/lib/node_modules/npm/lib/utils/error-handler.js:266:28)
npm ERR!     at /usr/local/lib/node_modules/npm/lib/update.js:27:20
npm ERR!     at /usr/local/lib/node_modules/npm/lib/outdated.js:33:30
npm ERR!     at cb (/usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
npm ERR!     at cb (/usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
npm ERR!     at /usr/local/lib/node_modules/npm/lib/outdated.js:100:22
npm ERR!     at cb (/usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
npm ERR!     at /usr/local/lib/node_modules/npm/lib/outdated.js:96:73
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:53:40
npm ERR!     at parseJson (/usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:82:32)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

npm ERR! System Darwin 12.4.0
npm ERR! command "node" "/usr/local/bin/npm" "update" "-g" "generator-server-configs"

/usr/local/lib/node_modules/npm/lib/utils/error-handler.js:266
  log.error("cwd", process.cwd())
                           ^
Error: EMFILE, too many open files
    at process.errorHandler (/usr/local/lib/node_modules/npm/lib/utils/error-handler.js:266:28)
    at process.EventEmitter.emit (events.js:117:20)
    at process._fatalException (node.js:272:26)
@dankohn dankohn referenced this issue in Medium/phantomjs Jun 17, 2013
Closed

Can't install on OS X #68

walesmd commented Jun 19, 2013

I came across this issue last night; as others have mentioned ulimit -n 4096 has taken care of the issue for now.

Relevant terminal activity:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

$ node -v
v0.10.12

$ npm -v
1.2.32


$ npm update -g
...
npm ERR! Error: EMFILE, too many open files
...
npm ERR! System Darwin 12.4.0
npm ERR! command "/usr/local/Cellar/node/0.10.12/bin/node" "/usr/local/bin/npm" "update" "-g"
npm ERR! cwd /Users/mike/Sites"
npm ERR! node -v v0.10.12
npm ERR! npm -v 1.2.32
npm ERR! syscall uv_cwd
npm ERR! code EMFILE
npm ERR! errno 20
...
Error: EMFILE, too many open files
    at process.errorHandler (/usr/local/Cellar/node/0.10.12/lib/node_modules/npm/lib/utils/error-handler.js:266:28)
    at process.EventEmitter.emit (events.js:117:20)
    at process._fatalException (node.js:272:26)

I just ran into this issue again with the following versions on OSX 10.8 with the default ulimit:

node -v # v0.10.12
npm -v # 1.2.32

Running the following command breaks it every single time:

npm update -g bower # same for npm update bower -g

Setting the ulimit to 4096 solves the issue, but as others have mentioned, this is not great. At the very least the error message could maybe mention this?

Pomax commented Jul 9, 2013

can this ticket be reopened, it looks like it's definitely not fixed yet (just ran into the same thing, found this thread, there's lots of comments after the initial closing that suggests this bug persists)

@edef1c edef1c reopened this Jul 9, 2013
@isaacs isaacs added a commit that referenced this issue Jul 10, 2013
@isaacs isaacs graceful-fs@1.2.3
Related to #3259

Because readFile and writeFile were not wrapped, there were cases where
a lot of package.json reading would cause problems.  This wasn't an
issue with earlier versions of graceful-fs, because it monkey-patched
the builtin FS module's open() method, so that readFile and writeFile
would Just Work.

However, that caused other problems.

Note that fs.ReadStream and fs.WriteStream are still not covered in the
open fd guard.  That can be done later, but this extends EMFILE coverage
to the more commonly used methods first.
e300730
Owner
isaacs commented Jul 10, 2013

Can one of you try running this and seeing if it fixes things for you? (Without the ulimit change, of course.)

npm i -g git://github.com/isaacs/npm#e300730

@isaacs This worked for me. (ulimit set to 256)
I double-checked with npm v1.2.32 and 1.3.2 (the bundled versions that come with 0.10.12 and 0.10.13) and both break. Installing your version works.

I tested with the command npm update -g bower.

Thanks for the fix. I hope this sorts it for everybody.

@isaacs Nope, I tried npm i -g git://github.com/isaacs/npm#e300730 but it's the same. And I have to reinstall npm after each try.

@ejeklint I had to run @isaacs command with sudo since the permissions of the bundled npm are different, so I ran into that problem as well.
Which command failed on your system? That way I could try on mine to see whether it's consistent.

For the record: the new version fixed the example I posted earlier (npm update -g bower).

@gillesruppert Yes I ran with sudo. Also tried a clean install of npm with entire /usr/local chmodded to myself.

It's npm update -g that always fails. Updating a single package works fine.

@ejeklint you are right: running npm update -g I get the dreaded EMFILE too many files open error with a ulimit of 256. As usual, a higher ulimit sorts it. /cc @isaacs

jsdnxx commented Jul 10, 2013

I am running into the same issues with EMFILE with the default ulimit of 256 on OSX / node 0.10.13 / npm 1.3.2 with a non-global dependency tree on one of my applications - was not receiving this with previous versions of node. npm ls works fine, but npm update or npm install from a clean node_modules directory fails.

Owner
isaacs commented Jul 11, 2013

Yeah, I can reproduce this. Rewriting graceful-fs to go back to monkey-patching the native fs module, since that's actually hte only thing that will work here.

Owner
isaacs commented Jul 11, 2013

Boy this was a doozy. I think I've got it worked out. Here's what was happening:

  1. The graceful-fs module sets up a queue system so that EMFILE failures will be retried.
  2. The lockfile module opens a lock file (and keeps it open -- stupid) in exclusive mode to prevent collisions.
  3. npm outdated adds packages to the cache (again, stupid) rather than simply checking which versions are available, so it does tons of tarball fetching and unpacking.

To address this problem:

  1. graceful-fs 2.0.0 again monkey-patches the native fs module. Even though monkey patching is gross, and I'm against it in principle, it's the only way to ensure that we catch every instance of a file open operation. Also, it's much simpler and less clever now, keeping a basic queue rather than attempting to count FDs, set a max, and detect the ulimit hit.
  2. lockfile 0.4.0 doesn't keep the files open. This means that it has to not pass the fd along, but that's fine. You don't want your lockfile to prevent you from doing things that you CAN do safely!
  3. Fixing the npm outdated behavior is a little bit more involved. I just added a feature to the registry in npm/npm-registry-couchapp@e20a430 to let you fetch urls like http://registry.npmjs.org/lockfile/~0.3.2 for returnign the maxSatisfying in a range. So, that should make the creation of a new registry client a bit simpler, since fetching a range is now just like fetching a normal version.

The next release of npm will have these features. Of course, you can pull from the latest git to pull it in now. It's working for me now even with a ulimit of 32 (albeit rather slowly, sometimes to the point where the fetches time out, but usually it will eventually finish).

@isaacs isaacs closed this Jul 11, 2013

Great news! Many thanks @isaacs for fixing this so fast.

Sounds great! 👍

Thanks @isaacs for the fix. Installing now.

Pomax commented Jul 15, 2013

Wow, that's a nice set of collaborative bugs. Great job finding that out! And props for fixing.

If anyone's still having issues rm -rf ~/.npm helped w/ the current brew install node --HEAD

@Soarez Soarez referenced this issue in gruntjs/grunt-cli Jul 25, 2013
Closed

Installation fails #27

Pomax commented Jul 29, 2013

Just so we know whether it landed with node 0.10.14/15, which tag would this have landed under? 1.3.4/5?

@tbosch tbosch added a commit to tbosch/angular.js that referenced this issue Mar 13, 2014
@tbosch tbosch chore(travis): fix version for grunt-cli and clear npm cache
Related to npm/npm#3259.
49d8914
@SimonKaluza SimonKaluza added a commit to InboxHealth/application_ember that referenced this issue Jun 10, 2015
@SimonKaluza SimonKaluza Added workaround for EMFILE error that occurs if tests are included i…
…n a build ( seems to be due to something similar to npm/npm#3259 ).
bf9e9c1

@tomfuertes hint just solved this (recurring?) issue for me. Thanks 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment