fs.writeFile hangs on high load (no error on v.0.6.6) #fs

[mrutid]

We are trying to serve as much requests as possible, and write one file per request. With small payload works fine, but on high payload scenarios while the synchronous version still working fine, the asynchronous version (fs.writeFile) hangs. Node stops serving requests and leave most of the files with 0-size (most of them neither get created).

There is no error in v0.6.6

In v.0.4.7 we got:

{ stack: [Getter/Setter], arguments: undefined, type: undefined, message: 'EMFILE, Too many open files \'aw_1031\'', errno: 24, code: 'EMFILE', path: 'aw_1031' }

Same behaviour in Ubuntu(VM) and Mac Osx.

This is the example script we are currently running with:

ab -n 30000 -c 500 http://HOST:8000/

var http = require('http');
var fs = require('fs');
var util = require ('util');
var i=0;

function writeALot(req, res){
    fs.writeFile("filetest"+i, "Just a try: "+i,
        function(err){
            if(err)    console.log(util.inspect(err));
        });
    i++;
    res.writeHead(200);
    res.end();
}
http.createServer(writeALot).listen(8000);

How can we manage the max number of concurrent fd? Any advice?

Thanks in advance.

[bnoordhuis]

Confirmed. It's a libuv bug and we're possibly also exhausting the libeio task queue.

You can avoid the worst of it by rewriting the request logic:

function writeALot(req, res){
    res.writeHead(200);
    fs.writeFile("filetest"+i, "Just a try: "+i,
        function(err){
            res.end();
            if(err)    console.log(util.inspect(err));
        });
    i++;
}

[mrutid]

Thanks!

The problem with the server logic is that is should work that way. Answering before processing (if it is possible), it's some kind of oneway relaying. If it is not possible we will slow down things a little bit.

Is it an already reported bug? any ticket to follow?

Thanks again : )

[bnoordhuis]

Is it an already reported bug?

It is now. :)

Set ulimit -n to some high number until this issue is resolved, that should mitigate it.

[mrutid]

I will try with https://github.com/isaacs/node-graceful-fs

If don't we will manage it "by hand".

+1 to node community

[bnoordhuis]

Continues in joyent/libuv#315.