Throw EMFILE errors

[bradparker]

Hi all,

Thanks so much for the awesome project. Sooo flexible, sooo easy to get up and running.

When reading in large projects (~10mb+) it seems that blowing out the ulimit causes metalsmith to fail silently. While I know I really should be splitting up such an big project, it took a while to track down the cause of the issue.

Let me know if I can provide any more info.

Cheers,
Brad

[ianstormtaylor]

Hmm damn that's definitely not good, not sure why it's failing silently

[Rob-ot]

I'm also having this problem with 1300 files totaling 6MB in the src directory.

Switching async.each to use eachSeries fixes the problem so it must just be reading too quickly. Switching it permanently might have performance implications (but if it's failing on 1300 now it shouldn't matter) so using async.eachLimit might be the best solution. I also have read about using graceful-fs in place of fs to fix the issue.

[kidwm]

Metalsmith · EMFILE, open '~/project/node_modules/metalsmith/node_modules/gnode/node_modules/regenerator/node_modules/defs/build/es5/jshint_globals/README'
This error stop me to use this configuration since there are too many files in node_modules folder.

{
  "source": "./",
  "plugins": {
    "metalsmith-ignore": "./node_modules/**/*"
  }
}

[ianstormtaylor]

Would accept a PR for this that limits the concurrency!

[tschaub]

Would accept a PR for this that limits the concurrency!

See #130.