http: EMFILE, Too many open files #793

Closed
wildchild opened this Issue Mar 16, 2011 · 14 comments

Comments

Projects
None yet
9 participants

The following code fails with "Too many open files". Just run it, change proxy settings for your browser and point it to one of the sites with millions of banners, do a number of refreshes.

var http = require('http');

http.createServer(function(request, response) {
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

Fails after a huge number of requests with:

net.js:695
        self.fd = socket(self.type);
                  ^
Error: EMFILE, Too many open files
    at net.js:695:19
    at dns.js:171:30
    at IOWatcher.callback (dns.js:53:15)

node 0.4.2 on OSX 10.6

isaacs commented Mar 16, 2011

Yes, you need to handle those.

Here's how I do it in npm:
https://github.com/isaacs/npm/blob/master/lib/utils/graceful-fs.js

On Wed, Mar 16, 2011 at 00:13, wildchild
reply@reply.github.com
wrote:

The following code fails with "Too many open files". Just run it, change proxy settings for your browser and point it to one of the sites with millions of banners, do a number of refreshes.

var http = require('http');

http.createServer(function(request, response) {
 var proxy = http.createClient(80, request.headers['host'])
 var proxy_request = proxy.request(request.method, request.url, request.headers);
 proxy_request.addListener('response', function (proxy_response) {
   proxy_response.addListener('data', function(chunk) {
     response.write(chunk, 'binary');
   });
   proxy_response.addListener('end', function() {
     response.end();
   });
   response.writeHead(proxy_response.statusCode, proxy_response.headers);
 });
 request.addListener('data', function(chunk) {
   proxy_request.write(chunk, 'binary');
 });
 request.addListener('end', function() {
   proxy_request.end();
 });
}).listen(8080);

Fails after a huge number of requests with:

net.js:695
       self.fd = socket(self.type);
                 ^
Error: EMFILE, Too many open files
   at net.js:695:19
   at dns.js:171:30
   at IOWatcher.callback (dns.js:53:15)

node 0.4.2 on OSX 10.6

Reply to this email directly or view it on GitHub:
joyent#793

ry commented Mar 16, 2011

we should probably have an async function for allocating a fd. reopening

ry commented Mar 16, 2011

for servers we just wait until an fd is available before accepting new connections - i think this is friendly.

for clients we can give them some amount of time during connect to allocate a fd - otherwise we fail by emitting 'error'. (in the same timeout that we wait for name resolution)

also warning should be printed to stdout that we're low on FDs like we do with servers.

I think we can handle disk FDs similarly?

weian commented Aug 1, 2011

I met the same problem whenever I start the server for more than 1 minuts, though I did not open too many connections.

I am using the following modules: sequelize, socket.io, and connect. I guess the problem is inside such modules, but do not have a way to test it. Could you provide any hint to help me figure out where the problem exists?

isaacs commented Aug 1, 2011

I've since spun graceful-fs off as its own library. https://github.com/isaacs/node-graceful-fs

It'd be very easy to repurpose this to node-core.

ry commented Aug 1, 2011

i'm of the opinion that we should fail fs ops on EMFILE

hynese commented Aug 17, 2011

@wildchild I'm using very similar code and am experiencing the exact same issue. Did you manage to resolve this EMFILE issue?

net_legacy.js:747
        self.fd = socket(self.type);
                  ^
Error: EMFILE, Too many open files
    at net_legacy.js:747:19
    at Object.lookup (dns_legacy.js:159:5)
    at Socket.connect (net_legacy.js:729:20)
    at Object.createConnection (net_legacy.js:268:5)
    at EventEmitter.connect (/home/hynese/Desktop/node_modules/dnode/index.js:42:26)
    at Function.connect (/home/hynese/Desktop/node_modules/dnode/index.js:262:22)
    at updateRemoteHash (/home/hynese/Desktop/server.js:86:9)
    at Socket.<anonymous> (/home/hynese/Desktop/server.js:39:8)
    at Socket.emit (events.js:67:17)
    at Socket._onReadable (net_legacy.js:684:14)

@hypense nope, just increased limits. But this is bad idea. I believe there should be a solution in core.

@ghost

ghost commented Nov 3, 2011

I'm getting this one too. If Isaac's solution were thought of as just "the node way of blocking in a situation where there isn't really any other choice", would that make it more core-worthy? Just asking ... not taking a position either way.

isaacs commented Nov 4, 2011

@sleeplessinc You can just use graceful-fs if you want to back off on fs EMFILE errors rather than abort. It's been working well for npm.

This shouldn't be in core.

@ghost

ghost commented Nov 4, 2011

Agreed. I think a note in the docs near the synch fs functions may be in order though, no?

Member

juliangruber commented Oct 21, 2012

but when using graceful-fs and another module uses plain fs you're screwed again

Owner

bnoordhuis commented Oct 21, 2012

Such is life. I think we forgot to close this issue. Closing then.

bnoordhuis closed this Oct 21, 2012

twolfson referenced this issue in Ensighten/spritesmith Mar 20, 2013

Closed

Fatal error: spawn EMFILE #11

I think We should re-open the issue.

graceful-fs does NOT solve it actually, yet it does wrap the error so the EMFILE is not emitted ;)...

i've been using both graceful and non-graceful, getting :error on readstream:Error: EMFILE, open
all the time. (with graceful its simply not accepting connections when this happens, but this error event is not emitted)

found n steady way to reproduce the error:
accessing the same file more then 1000 times, evokes it.

using centos linux / nodejs 10.0.15

joyent#6041

isaacs/node-graceful-fs#20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment