A static file generator for read only web services. Currently supports MySQL backend only.
Python
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src/dumper
.gitignore
README.textile
fabfile.py
requirements.txt
setup.py

README.textile

Dumper is a static file generator for building simple read only web services. Think Jekyll or Nanoc but for XML and JSON.

Still to do:

  • bundling commands for deploying changes
  • build in web server for testing
  • versioning
  • date based snapshots
  • support for more output formats than JSON and XML
  • support for more backends than just MySQL
  • logging other than command line output

Trying Dumper

First you’ll need a MySQL database. I’ve provided a simple table definition and some data to get you started. See the tests folder for an example sql file.

mysqladmin create dumper -u root
mysql dumper < people.sql -u root

You can now install dumper using the provided setup.py file. This should install all the requirements. I’d recommend using virtualenv to sandbox your python environment.

python setup.py install

You should now have the dumper command line application available on your path. Dumper uses a configuration file to determine which backend to use. An example configuration file is shown below:

[Dumper]
path: people
index: id
backend: mysql

[Database]
sql: SELECT id, name FROM people
host: localhost
username: root
password:
database: dumper

Using the example file from the tests directory like so:

dumper -c people.truck dump

This should give you something like this output to the console:

[Created] output/people
[Created] output/people/1.json
[Created] output/people/1.xml
[Created] output/people/2.json
[Created] output/people/2.xml
[Created] output/people.json
[Created] output/people.xml

Run dumper help for a full list of options.

Processors

Processors allow you to massage the data taken from the SQL query or to modify the XML output to match your preferred schema. You specify processors in the ini file like so:

[Dumper]
path: people
index: id
backend: mysql
post_processors: examplepre
pre_processors: examplepost

These are simply python modules on your python path. Note that the directory where you run dumper is added to the python path to make it easier. Processors run one after the other, passing the output of the first to the second and so on. Preprocessors take the python list or dictionary with the results of the SQL query. Postprocessors take the XML string before it is written to disk. A simple processor might just delete all XML output like so:

def processor(input):
    return ""

More likely you’ll want to do validation on data based on an external, non-sql source, or you’ll want to transform the XML to match a specific output format.

Subclassing

But what if you want complete control over how things are serialized. Or even how things are output? You can register your own subclasses as backends and use those.

The following is just a noddy example. With this class available you can use the backend ‘mysql2’ and it will work just the same as the ‘mysql’ backend.

from dumper import BaseDumper, MySQLDumper

class MySQLDumper2(MySQLDumper, BaseDumper):
    @classmethod
    def is_dumper_for(cls, type):
	"The string passed in to the factory to get this class"
	return type == 'mysql2'

In order for the backend detection to work you need to get these classes imported. For that we have the import parameter in the config file which allows for a comma seperated list of import commands.