Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Render documents as we need them #629

Open
balupton opened this Issue Sep 3, 2013 · 11 comments

Comments

Owner

balupton commented Sep 3, 2013

Rather than rendering everything in one initial batch, then rendering everything that is affected by a chance in subsequent batches, perhaps instead we can re-render things only when needed (or rather when requested).

Doing this could be as simple as setting everything in your database as dynamic: true but then you lose all performance benefits as nothing is cached.

It seems a hybrid here would be nice, current idea is:

  1. We wait until a request happens, we then render the document to the file system like normal.
  2. When a change happens we take note of it, but only re-render affected documents when they get requested.
  3. We index files, but only load them prior to render.

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

Owner

balupton commented Sep 3, 2013

#336 while not required would help performance of this dramatically

Contributor

greduan commented Sep 3, 2013

So basically a file isn't rendered until requested?

Owner

balupton commented Sep 3, 2013

yep

Contributor

greduan commented Sep 3, 2013

Well... That'll certainly make the initial regeneration faster but it'll slow down testing and such I think.

I think we should perhaps give this a try in an update, see how it works, and if it doesn't work as expected perhaps make it a setting?

I'm reluctant about it, but perhaps it'll be better than I expect. :)

gebrits commented Sep 3, 2013

If I'm reading this correctly: +1
Not sure if I do though :)

Would this match well with my braindump style comment on "Discussion:
Rewrite in Noflo?
bevry#600

"Major thing to think about imo would be the granularity of "information
packets" in the flow. Currently, transitions likegenerate, write, etc. are
visited with batches of documents at a time afaik. Loads of additional
use-cases would open up if an 'information packet' could instead be a
single document. For one, this would mean regeneration of files would be
constrained to those that are directly influenced by a change of said
single document. I'm completely unsure atm if this is at all feasible btw,
but I think it's worth thinking about"

Owner

balupton commented Sep 3, 2013

@greduan yeah, this will take a long time to implement as there are a lot of things to take into consideration.

It is necessary however for DocPad to handle huge databases (say 30,000+ files with a few gigabytes of data), as doing everything at once is impossible due to memory and performance constraints.

The other thing that makes this request so awesome is that it would also accomplish #590 automatically. Imported tumblr documents could just be added to the database in the background, and then rendered (along with things like content listings) when required. We would need to think about the performance hit for things like paging here.

@gebrits sounds more or less the same to this issue, with the less being the technical side of things. Not yet sure what the best way to accomplish this technically within DocPad would be.

Contributor

greduan commented Sep 3, 2013

Ah OK. That seems like a good reason to go for that then. :)

Well I don't mind doing this if it's for performance reasons. However perhaps it would be wise to find a balance between what should be in the initial render and what shouldn't.

For that to happen though performance tests and a general vote by users will be necessary I think.

gebrits commented Sep 3, 2013

I was a bit verbose, but basically what I suggest it to have every
possible stage in the 'pipeline' be requested (i.e pull) or fire based on
rules (push) and granular to a particular document. So not just the
'render' stage.

This would enable all kinds of extra cool things: reacting to changes from
external Gui's or Zapier webhook listener, etc. Updating other custom
'consumers' on a document-level, which can be fed before the 'render'
stage: for example reindex the updated document in something like
solr/elasticsearch, etc.

Of course you may already have a pretty good feel on where you want to go,
but I just wanted to share my 2 cents :)

On Tue, Sep 3, 2013 at 8:40 PM, Eduán Lávaque notifications@github.comwrote:

Ah OK. That seems like a good reason to go for that then. :)

Well I don't mind doing this if it's for performance reasons. However
perhaps it would be wise to find a balance between what should be in the
initial render and what shouldn't.

For that to happen though performance tests and a general vote by users
will be necessary I think.


Reply to this email directly or view it on GitHubhttps://github.com/bevry/docpad/issues/629#issuecomment-23736582
.

Owner

balupton commented Sep 7, 2013

This will be up next for me.

@ghost ghost assigned balupton Sep 7, 2013

Owner

balupton commented Sep 27, 2013

This is put back on hold. We need to figure out how to do this, without impacting other plugins.

For instance, right now the generate events assume that it applies to everything. Where with this, it wouldn't be the case.

Contributor

pflannery commented Oct 28, 2013

what would be nice is if we could determine how files are rendered by environment and\or "docpad command"

i.e. document meta could have this

---
title: "I only get rendered in development"
render false
environment:
    development:
        render: true
---

or

---
title: "I only get rendered in docpad generate and only served during docpad server"
url: "/path/fancy-page.html"
docpad-command:
    generate:
        render: true
    server:
        render: false
        write: false
---

Obviously setting ever file would be a chore but we could write queries during a suitable docpad event to set this kind if meta in a batch

This could help keep overhead down when running docpad server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment