Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exporters / Sources #597

gebrits opened this issue Aug 13, 2013 · 3 comments

Exporters / Sources #597

gebrits opened this issue Aug 13, 2013 · 3 comments


Copy link

@gebrits gebrits commented Aug 13, 2013

See #597 (comment) for latest plan.

Would it be useful to have a 'generic export hook / command' that allows you to loop all documents as they exist just before they would be written to out (but without doing the actual writing)?

I imagine this to be a separate docpad cli command, say docpad export, which parameterized could run your export of choice.

particularly: I wanted a way to quickly generate the accurate set of meta-data properties of all documents. Since these meta-data properties may be enriched by all kinds of hooks there needs to be a reliable way to process the documents as they normally would. I.e: all hooks capable of changing the meta-data when doing a docpad generate would need to run when doing the docpad export

I've quickly hacked the following (a separate node project which includes docpad) which seems to run most (?) of the needed hooks before it loops the documents. It seems to run okay and obviously is blistering fast since if skips all the actual writes:

// Add DocPad to our Application
var docpadInstance,
_ = require("lodash");

var docpadInstanceConfiguration = {
    env: 'static'

var balUtil = require('bal-util'),
extractOptsAndCallback = require('extract-opts').extractOptsAndCallback;

 docpadInstance = require('docpad').createInstance(docpadInstanceConfiguration,     function(err){
if (err)  return console.log(err.stack);

var that = docpadInstance;

var generateRender = function(opts, next) {
    var docpad, _ref1;
    _ref1 = extractOptsAndCallback(opts, next), opts = _ref1[0], next = _ref1[1];
    docpad = that;
    opts.templateData || (opts.templateData = that.getTemplateData());
    opts.renderPasses || (opts.renderPasses = that.getConfig().renderPasses);
      object: docpad,
      //action: 'contextualizeFiles renderFiles writeFiles',
      action: 'contextualizeFiles',
      args: [opts],
      next: function(err) {
        return next(err);
    return that;

    docpadInstance.generateRender = generateRender;

    docpadInstance.action('generate', function(err){
        throw new err;
    _.each(docpadInstance.getCollection('html').models, function(doc){
        //do stuff

I think it wouldn't require that much work to make this into something more robust. Would there be any interest for such a feature?

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

Copy link

@Delapouite Delapouite commented Aug 14, 2013

Yes. There should be a way to dump the queryEngine in memory database to a persistent database.
Also the writing part should be optional.

In many of my use cases I just want to perform queries on the generated database. Parsing the documents files and writing them in /out are loooong steps that could be avoided.

Copy link

@balupton balupton commented Sep 26, 2013

The way we have planned to get this working is in a few parts:

The first was getting importers done, that's now completed.

The second is turning the file system interactions into just another importer/exporter. This was reveleaved a bit in the architeture vision here - #543 - but not covered in detail.

Essentially, what we need to do is move away from the current class & code based structure of the models, and turn them more into data stores and event mediators that plugins can then subscribe to.

Support for additional importers/exporters - perhaps we should just call them datastores instead! - would make this a lot easier.

One thing that needs to happen for this to be the case is we also need to allow better support for background rendering, or rather just in time rendering, or rather as you need it rendering. Right now rendering everything in one go is good for static site generation, but not good if you are running a dynamic server which is becoming an increasing use case.

Finally, we would need to add support for globally installed plugins, but that is simple enough. E.g. npm install -g docpad-plugin-fs to install the file system plugin for all projects.

@balupton balupton mentioned this issue Sep 26, 2013
12 tasks
@balupton balupton changed the title Exporters Exporters / Sources Mar 21, 2014
Copy link

@balupton balupton commented Mar 21, 2015

This approach no longer makes sense. See here for the latest thoughts:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

3 participants