Better Importer support #500

balupton opened this Issue May 6, 2013 · 28 comments



balupton commented May 6, 2013

To see how Importers fit into the DocPad Architecture, check out #543


To accomplish our army of importer support and to get the syte skeleton operational we need to accomplish the following things:

Optional tasks:

  • allow plugins to take control of reading and writing files
  • abstract out read and write interfaces into user land
  • abstract out watch interface into user land
  • allow support for bundled plugins
  • better translation support
  • better logging support inside file class


Importers like JSON, MongoDB, Dropbox, Tumblr, RSS

balupton was assigned May 6, 2013

balupton referenced this issue in docpad/docpad-skeleton-syte Jun 12, 2013


REQUEST - facebok feed / wall #3

easy support for injecting meta data into documents -> minicms or Collections Editor ?


balupton commented Jun 13, 2013

@dospuntocero currently injecting meta data via an API is a huge pain, due to the requirement of a data field that does not exist on virtual documents. E.g.

# Comment data
commentData = """
    title: "#{commentTitle}"
    for: "#{commentFor}"
    author: "#{commentAuthor}"
    layout: "comment"
    date: "#{date.toISOString()}"


When really having an API like: docpad.createDocument(meta: {a,b,c,}, body: '...') would be a lot more useful.

In turn this functionality will come hand in hand with the addition of virtual document support, allowing us to have documents that don't have physical locations. Allowing us to actually say have tumblr articles directly inside our database, allowing you do so something like @getCollection('tumblr').

This also makes it way easier for MiniCMS and DCE to do their magic, as right now they have to use the same hacks as above and those of the native comments plugin

+1 for that. it sounds interesting

I'm waiting for this!!!


balupton commented Jun 29, 2013

I've completed the groundwork for this on the dev branch and got the docpad tests passing and most of the plugin tests. The paged plugin tests failed, so will need to look into that.

Next up is working on the injection side of things.


balupton commented Jun 29, 2013

@benjamind would you be able to look into the failing paged plugin tests by any chance?

Just pulled down latest docpad, and docpad-plugin-paged and run the tests and it doesn't look like there's any failures?


balupton commented Jun 30, 2013

Did you do the dev branch?

git clone blah blah
cd docpad
git checkout dev
npm install
cake compile
npm link
cd paged plugin
npm link docpad
cake compile
cake test

balupton commented Jul 1, 2013

@benjamid I've started work on debugging

@balupton sorry I've not been able to put more time into this. Got myself completely overloaded at the moment just trying to keep my head above water!

balupton referenced this issue in docpad/docpad-plugin-paged Jul 1, 2013


Get it working with the new importers work #8


balupton commented Jul 1, 2013

@benjamind no worries, lets continue the paged plugin side of things here: docpad/docpad-plugin-paged#8


balupton commented Jul 2, 2013

v6.44 is now out with virtual document support.

@cauld and @jeremyfa you will want to check your admin plugins for v6.44 support as a lot has changed, refer to the paged plugin for reference on how some of the new stuff works.


balupton commented Jul 2, 2013

Still got a bit to go before importers are ready, but boy are we getting close! :)

Next up is injecting clean/new/fresh documents. So far the paged document just clones existing ones. So fresh documents will be the next challenge.

jeremyfa commented Jul 2, 2013

Thanks for the update! I will check in the following days. Note that for now the MiniCMS plugin is just writing and reading files using regular file I/O, but be sure that once Docpad is exposing a clean API to manage files and documents I will update the plugin.


balupton commented Jul 2, 2013

@jeremyfa great :) Looking forward to it :)


balupton commented Jul 2, 2013

Also, I've done up a quick video to explain the importers feature and how it ties into everything:

kr1zmo commented Jul 3, 2013

My article on DocPad is completed; please if anyone finds spelling or grammar mistakes or other errors, tell me immediately.


balupton commented Jul 3, 2013

@kr1zmo looks great! couldn't find any typos from my read, all great stuff :)

kr1zmo commented Jul 3, 2013

Hey thanks Ben, I enjoyed writing this over the past day. Fun stuff.

We Would like to join the importer work and release some plugins for instagram, twitter & youtube. Is there any code to work with ? Any plan when importer architecture will be released into docpad?

balupton referenced this issue in docpad/docpad-plugin-sass Jul 10, 2013


Add support for sourcemaps #14


balupton commented Jul 10, 2013

@weareburo sweet! cool that you're keen to jump on the bandwagon!

For timeframe, soon is a good estimate :-) Importers are our current focus - plus I'm working with a few others to help make it happen which is exciting! I'll keep updating this as progress is made, and so you can try our and provide your feedback on the latest stuff.

If you want, if you could do a list of things you would like to see, or have implemented, we can figure out what tasks make sense for us to do, and which make sense for you to do if you want, as well as figure out which order of tackling them would be most helpful. Should give you something to run with right now to help speed things up.

Also, if you're able any dontations to will help speed things up no matter how small or large ;-) Thanks!


balupton commented Jul 17, 2013

All tests passing and code review complete for dev branch and tumblr plugin, left to do before new release:

  • writeSource meta data
  • tumblr user based meta data insertion
  • documentation
  • native comments plugin
  • syte skeleton

great benjamin, is this one is the implementation or should i wait for the release to start work on importers.


balupton commented Jul 18, 2013

It's that one, but wait for the release, it'll be pretty soon 👍

@balupton balupton added a commit to docpad/docpad-plugin-tumblr that referenced this issue Jul 18, 2013

@balupton balupton Working with latest importer code - docpad/docpad#500 f2ca1d9

balupton commented Jul 19, 2013

We've actually just hit a hiccup with the syte skeleton implementation of the tumblr plugin. As layouts are the way for rendering tumblr data, if you do contentRenderedWithoutLayouts then the tumblr content is not wrapped correctly as the essential layout for that is not included.

To conquer this, we either need to think of a way that doesn't use layouts, or add a meta data property to the layouts to say includeInContentRenderedWithoutLayouts: true which is pretty lame but should work well. Maybe includeLayoutWithinContent is more concise.

balupton closed this Jul 23, 2013


balupton commented Sep 26, 2013

Exporters discussion continued here #597


balupton commented Mar 21, 2015

This approach no longer makes sense. See here for the latest thoughts:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment