Broken on large number of articles

[webhive]

I did a stress test for wintersmith to estimate it's work on large number of articles. I am care about to how such kind of engines will behave after couple of years of blogging. Dont want to frustrate with it :) I did a tests for DocPad and for Hexo already, so now my victim is wintersmith.

So after about 800 files added (I was used another-test/ article as template) I got error

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: EMFILE, open '<path to test>/build/articles/test-7-36/index.html'
Command exited with non-zero status 8

After this there are no destination files generated - only empty folders.

wintersmith preview

work well without any problems

So there is obvious issue ... may be with too many files open or something like this, but anyway this is a sufficient limit for this engine.

[jnordberg]

Yeah, i have a todo to actually respect the file limit everywhere, i think it recurses at some point. Try adding the _fileLimit key to your config.json, the default is 40. Something like 10 might give you better results.

[webhive]

It changed nothing - still broken after about 800 posts

[webmasterkai]

What about using https://github.com/isaacs/node-graceful-fs ?

Something like

try {
// optional dependency
fs = require("graceful-fs")
} catch (er) {
fs = require("fs")
}

[tlupfer]

It's OS specific. I have around 750 posts and get the same error on my macbook (fixed with ulimit -n 16384 ... default open files per process is 256), but do not in linux. @webmasterkai might have the better solution though.

[jeromew]

@webhive keep us posted on your research and your list of wintersmith pros/cons.

I hope #207 PR fixes your file descriptor issue. it fixes the EMFILE bug for me with 800+ posts.

[webhive]

Hmm - unfortunately no changes - got exatly the same error after same number of posts written.

$ wintersmith -V
2.0.10-pre

Here is a gist about it https://gist.github.com/webhive/8237326
So anyone can do the same test

[webhive]

@webhive keep us posted on your research and your list of wintersmith pros/cons.

I could be glad to say something about wintersmith pros, but can't do it .... guess why? :)

About contra - after new wintermith project generated I got jade error about deprecated syntax in some templates so wasted some time to fixing it. And finally after got issue on benchmark honestly I lost an interest to this project. This is one big frustration.

The only single thing why I am still here is rather quick community replies and (I am really suprised) quick attempt to fix it. I am really appreciate it so at least will try to help to resolve this issue and after it I will say something pro :)

Update:

Just uploaded to gist my benchmarking diagram so wintersmith looks realy good in comparison with other similar projects. So this is my pro about it.

[jeromew]

@webhive thank you for your time on this.

before I test your gist, can we confirm that you tested wintersmith with my modification ? I do not have commit access to wintersmith so we need to wait until @jnordberg reviews and commits my modification to the project before you can test it with the wintersmith that you download at https://github.com/jnordberg/wintersmith

to be sure you have wintersmith with my patch (until it is accepted upstream) you can use this sequence of commands

 wget https://github.com/jeromew/wintersmith/archive/lots-of-files.zip
 unzip lots-of-files.zip
 cd wintersmith-lots-of-files
 npm install
 coffee -o lib/ -cw src/
 cd examples/blog
 npm install
 ../../bin/wintersmith build

(you will need the "coffee" command in order to compile coffeescript,so add a "npm install -g coffee-script" at the beginning of the sequence if you do not have it)

you will have to launch your benchmark on this version of wintersmith which is exactly the same as 2.0.10-pre except for the modifications on jeromew@d0b2e16

regading the jade deprecation warnings, jade was upgraded over christmas by the jade team (0.35 => 1.0.2).
the wintersmith package.json has a dependency on jade ">=0.25.0" so this is why a new install gets the new jade and the deprecation warnings. Until a patch is commited, for the blog example you can edit templates/index.jade

replace

!{ typogr(article.intro).typogrify() }

by

| !{ typogr(article.intro).typogrify() }

(just add the "| " before the current expression - not at the beginning of the line, but right before the "!")

[webhive]

can we confirm that you tested wintersmith with my modification ? I do not have commit access to wintersmith so we need to wait until @jnordberg reviews and commits my modification

Definitely no. I just pulled master branch from repository. My previous version was 2.0.9 after update 2.0.10-pre, so I desided your changes was included. Now I see - not.

regading the jade deprecation warnings

Already fixed it manually - it was not something blocking me - just looks frustrating. Instead of 'just work' - I got system what even was not able to start :( And btw another fix required was replace in layout file !!! 5 to doctype html, but seems it fixed in 2.0.10

you can use this sequence of commands

Thank you, but not sure will do it today. I am afraid I will return online after Jan 9

[webhive]

Confirm - patches from @jeromew resolved issue.
Got it working well without problems.