spawn EMFILE

[mkrech]

Hi,
i want to import 20000 Document files in node.js.

after 300 exec calls I get:

shell.js: internal error
Error: spawn EMFILE
at errnoException (child_process.js:980:11)
at ChildProcess.spawn (child_process.js:927:11)
at exports.spawn (child_process.js:715:9)
at Object.exports.execFile (child_process.js:607:15)
at Object.exports.exec (child_process.js:578:18)
at execSync (/var/www/filesearch/node_modules/shelljs/shell.js:1793:9)
at _exec (/var/www/filesearch/node_modules/shelljs/shell.js:1130:12)
at /var/www/filesearch/node_modules/shelljs/shell.js:1487:23
at FileScanner.ingestDB (/var/www/filesearch/crawl/filescanner.js:41:25)
at callbacks (/var/www/filesearch/node_modules/express/lib/router/index.js:161:37)

I tried it asynchon or synchron. It seems fpr me that the file handle in exec is not closed.

ulimit -n xxx doesn't help here ..

You can reproduce this behaviour with following snippet:

    for(i=0;i<1000;i++) {
        var child = exec('node -v', {async:false});
        console.log(i);
    }

TIA
Michael

[stephenmathieson]

you're likely to get all sorts of errors opening 20,000 files at once...

[jfsiii]

@arturadib Have you considered using https://github.com/isaacs/node-graceful-fs instead of fs? https://github.com/isaacs/node-graceful-fs#improvements-over-fs-module lists attempts to avoid EMFILE errors as a feature.

[fasiha]

Any simple work-around?

Edit: on Mac OS X, ulimit -n 10000 in the same shell as node worked per thread on nodejs list.

[nfischer]

Closing this. It looks like a user issue, not a ShellJS issue. If someone can't find a suitable workaround, they can try to use graceful-fs as a global patch of the fs module. This is not officially supported, nor has it been tested.