Still to do:
- bundling commands for deploying changes
- build in web server for testing
- date based snapshots
- support for more output formats than JSON and XML
- support for more backends than just MySQL
- logging other than command line output
First you’ll need a MySQL database. I’ve provided a simple table definition and some data to get you started. See the tests folder for an example sql file.
mysqladmin create dumper -u root mysql dumper < people.sql -u root
You can now install dumper using the provided setup.py file. This should install all the requirements. I’d recommend using virtualenv to sandbox your python environment.
python setup.py install
You should now have the dumper command line application available on your path. Dumper uses a configuration file to determine which backend to use. An example configuration file is shown below:
[Dumper] path: people index: id backend: mysql [Database] sql: SELECT id, name FROM people host: localhost username: root password: database: dumper
Using the example file from the tests directory like so:
dumper -c people.truck dump
This should give you something like this output to the console:
[Created] output/people [Created] output/people/1.json [Created] output/people/1.xml [Created] output/people/2.json [Created] output/people/2.xml [Created] output/people.json [Created] output/people.xml
dumper help for a full list of options.
Processors allow you to massage the data taken from the SQL query or to modify the XML output to match your preferred schema. You specify processors in the ini file like so:
[Dumper] path: people index: id backend: mysql post_processors: examplepre pre_processors: examplepost
These are simply python modules on your python path. Note that the directory where you run dumper is added to the python path to make it easier. Processors run one after the other, passing the output of the first to the second and so on. Preprocessors take the python list or dictionary with the results of the SQL query. Postprocessors take the XML string before it is written to disk. A simple processor might just delete all XML output like so:
def processor(input): return ""
More likely you’ll want to do validation on data based on an external, non-sql source, or you’ll want to transform the XML to match a specific output format.
But what if you want complete control over how things are serialized. Or even how things are output? You can register your own subclasses as backends and use those.
The following is just a noddy example. With this class available you can use the backend ‘mysql2’ and it will work just the same as the ‘mysql’ backend.
from dumper import BaseDumper, MySQLDumper class MySQLDumper2(MySQLDumper, BaseDumper): @classmethod def is_dumper_for(cls, type): "The string passed in to the factory to get this class" return type == 'mysql2'
In order for the backend detection to work you need to get these classes imported. For that we have the import parameter in the config file which allows for a comma seperated list of import commands.