Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

docpad fails to generate files for large size websites #446

Closed
segalr3 opened this Issue Mar 4, 2013 · 9 comments

Comments

Projects
None yet
5 participants

segalr3 commented Mar 4, 2013

docpad version = 6.21.10
node version = v0.8.18
npm version = 1.2.7
Ubuntu 12.04

What i did:

I have a website with many languages, which i am migrating to docpad.
First i tried to generate for 2 languages, all was generated fine.
Then i copied the rest of the languages folders to the documents directory and tried to generate again.

what happend:

I actually hit the nodejs memory limit:

"FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory"

no files were created in the out directory.

Please advise


Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

segalr3 commented Mar 4, 2013

I used a few partials in my layouts, after removing the partials, docpad managed to generate all the files.
So the issue is having partials and lots of documents...

Owner

balupton commented Mar 4, 2013

Ref bevry#276

Researching static site generators and noticed this issue. Out of curiosity @segalr3, would you happen to remember how many was a lot? Hundreds, thousands or millions?

I'm not seeing any limitations or guidelines related to this mentioned in the docs.

segalr3 commented Nov 22, 2014

Hundreds if i recall correctly, it was a multilingual site with ~30 languages.
I ended up using grunt to create a docpad instance for each language, generate it and kill the instance. If i tried to generate the whole site it would crash after a few minutes.

Sent from my iPhone

On 22 בנוב׳ 2014, at 01:20, cory notifications@github.com wrote:

Researching static site generators and noticed this issue. Out of curiosity @segalr3, would you happen to remember how many was a lot? Hundreds, thousands or millions?

I'm not seeing any limitations or guidelines related to this mentioned in the docs.


Reply to this email directly or view it on GitHub.

Owner

mikeumus commented Nov 22, 2014

Hey Gang, @dorajistyle just posted this as example of handling many files with DocPad:
https://github.com/dorajistyle/blogger-docpad
(in this pull-req: docpad/documentation#147)

Check it out, hope it helps. 😃👍

Thanks. Very helpful.
On Nov 21, 2014 9:36 PM, "Ram Segal" notifications@github.com wrote:

Hundreds if i recall correctly, it was a multilingual site with ~30
languages.
I ended up using grunt to create a docpad instance for each language,
generate it and kill the instance. If i tried to generate the whole site it
would crash after a few minutes.

Sent from my iPhone

On 22 בנוב׳ 2014, at 01:20, cory notifications@github.com wrote:

Researching static site generators and noticed this issue. Out of
curiosity @segalr3, would you happen to remember how many was a lot?
Hundreds, thousands or millions?

I'm not seeing any limitations or guidelines related to this mentioned
in the docs.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub
#446 (comment).

I sent pull request to jekyll-import but not merged yet.

You can download importer 'blogger_docpad.rb' below:

https://github.com/dorajistyle/jekyll-import/blob/master/lib/jekyll-import/importers/blogger_docpad.rb

This is good info, but I'll be needing thousands of pages. Maybe add a limitations section in the docs with this info? This is a pretty big one IMO.

Owner

balupton commented Mar 21, 2015

The raw plugin solves this if most of your files are static files. However, if they are documents, it gets more interesting.

Moving the discussion here: https://discuss.bevry.me/t/docpad-is-slow-for-large-websites/47

@balupton balupton closed this Mar 21, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment