http: EMFILE, Too many open files

[wildchild]

The following code fails with "Too many open files". Just run it, change proxy settings for your browser and point it to one of the sites with millions of banners, do a number of refreshes.

var http = require('http');

http.createServer(function(request, response) {
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

Fails after a huge number of requests with:

net.js:695
        self.fd = socket(self.type);
                  ^
Error: EMFILE, Too many open files
    at net.js:695:19
    at dns.js:171:30
    at IOWatcher.callback (dns.js:53:15)

node 0.4.2 on OSX 10.6

[isaacs]

Yes, you need to handle those.

Here's how I do it in npm:
https://github.com/isaacs/npm/blob/master/lib/utils/graceful-fs.js

On Wed, Mar 16, 2011 at 00:13, wildchild
reply@reply.github.com
wrote:

The following code fails with "Too many open files". Just run it, change proxy settings for your browser and point it to one of the sites with millions of banners, do a number of refreshes.

var http = require('http');

http.createServer(function(request, response) {
 var proxy = http.createClient(80, request.headers['host'])
 var proxy_request = proxy.request(request.method, request.url, request.headers);
 proxy_request.addListener('response', function (proxy_response) {
   proxy_response.addListener('data', function(chunk) {
     response.write(chunk, 'binary');
   });
   proxy_response.addListener('end', function() {
     response.end();
   });
   response.writeHead(proxy_response.statusCode, proxy_response.headers);
 });
 request.addListener('data', function(chunk) {
   proxy_request.write(chunk, 'binary');
 });
 request.addListener('end', function() {
   proxy_request.end();
 });
}).listen(8080);

Fails after a huge number of requests with:

net.js:695
       self.fd = socket(self.type);
                 ^
Error: EMFILE, Too many open files
   at net.js:695:19
   at dns.js:171:30
   at IOWatcher.callback (dns.js:53:15)

node 0.4.2 on OSX 10.6

Reply to this email directly or view it on GitHub:
#793

[ry]

we should probably have an async function for allocating a fd. reopening

[ry]

for servers we just wait until an fd is available before accepting new connections - i think this is friendly.

for clients we can give them some amount of time during connect to allocate a fd - otherwise we fail by emitting 'error'. (in the same timeout that we wait for name resolution)

also warning should be printed to stdout that we're low on FDs like we do with servers.

I think we can handle disk FDs similarly?

[weian]

I met the same problem whenever I start the server for more than 1 minuts, though I did not open too many connections.

I am using the following modules: sequelize, socket.io, and connect. I guess the problem is inside such modules, but do not have a way to test it. Could you provide any hint to help me figure out where the problem exists?

[isaacs]

I've since spun graceful-fs off as its own library. https://github.com/isaacs/node-graceful-fs

It'd be very easy to repurpose this to node-core.

[ry]

i'm of the opinion that we should fail fs ops on EMFILE

[ghost]

@wildchild I'm using very similar code and am experiencing the exact same issue. Did you manage to resolve this EMFILE issue?

net_legacy.js:747
        self.fd = socket(self.type);
                  ^
Error: EMFILE, Too many open files
    at net_legacy.js:747:19
    at Object.lookup (dns_legacy.js:159:5)
    at Socket.connect (net_legacy.js:729:20)
    at Object.createConnection (net_legacy.js:268:5)
    at EventEmitter.connect (/home/hynese/Desktop/node_modules/dnode/index.js:42:26)
    at Function.connect (/home/hynese/Desktop/node_modules/dnode/index.js:262:22)
    at updateRemoteHash (/home/hynese/Desktop/server.js:86:9)
    at Socket.<anonymous> (/home/hynese/Desktop/server.js:39:8)
    at Socket.emit (events.js:67:17)
    at Socket._onReadable (net_legacy.js:684:14)

[wildchild]

@hypense nope, just increased limits. But this is bad idea. I believe there should be a solution in core.

[sleeplessinc]

I'm getting this one too. If Isaac's solution were thought of as just "the node way of blocking in a situation where there isn't really any other choice", would that make it more core-worthy? Just asking ... not taking a position either way.

[isaacs]

@sleeplessinc You can just use graceful-fs if you want to back off on fs EMFILE errors rather than abort. It's been working well for npm.

This shouldn't be in core.

[sleeplessinc]

Agreed. I think a note in the docs near the synch fs functions may be in order though, no?

[juliangruber]

but when using graceful-fs and another module uses plain fs you're screwed again

[bnoordhuis]

Such is life. I think we forgot to close this issue. Closing then.

[syberkitten]

I think We should re-open the issue.

graceful-fs does NOT solve it actually, yet it does wrap the error so the EMFILE is not emitted ;)...

i've been using both graceful and non-graceful, getting :error on readstream:Error: EMFILE, open
all the time. (with graceful its simply not accepting connections when this happens, but this error event is not emitted)

found n steady way to reproduce the error:
accessing the same file more then 1000 times, evokes it.

using centos linux / nodejs 10.0.15

#6041

isaacs/node-graceful-fs#20