Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Datamith is a MySQL database conversion/import/fusion framework.
Branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.


= Datamith

Datamith is a MySQL database conversion/import/fusion framework. To use it, you will have to define some rules in the <tt>tables/</tt> directory.

A rule is a class derivated from Datamith::Converter. It can :

* simply transfert data
* convert charset ( using iconv )
* update data if a record already exists
* take the value of one field and load it in an other field
* do some basic type conversion between datetime and timestamp
* append data instead of preserving the primary key if append mode is set ( tweakable on a per record basis with a Proc conditionnal )
* remember the old primary value and the new one to keep track of the associations when in append mode

There is one rule file per table to convert, and each set of rules is lauched individually against each row. A rule file and the class it contains must be named accordingly. The filename can be prefixed with a number to set the order.

== Configuration and rake tasks

You should set the hosts, users, passwords and databases in config.yml.

As soon it is done, you're given some rake tasks to generate the rule files.

Generating a file for all the tables :

  rake tables:populate

Generating a file for a specific table : ( assuming there is an "users" table )

  rake tables:generate:users

You may also set in this file the "tables_to_convert" key. I should be a
list of the tables you want to be generated. The list order is preserved, so
<tt>rake tables:populate</tt> with :

  tables_to_convert: [ users, posts, comments ]

will generate :


No list or empty list means all the tables.

== Execution

When you're done with your rules, simply launch :
  rake convert

If you prefer a dump as a sql file, you can use instead :
  rake convert:dump

( You are responsible of the output redirection )

== Conversion

You will find more informations in the documentation for Datamith::Converter .
Here is a example of what can be done :

  # tables/10_User.rb
  class User < Converter
    @@old_table = "xy_user"
    @@new_table = "users"
    @@old_primary_key = :account_number
    @@skip = %w( 1 2 )
    @@charset_from = "latin1"

    def run
      append :condition => { |old,new| ( 15..20 ).include?( old[ :account_number ] ) }
      convert :int, :account_number, :id
      convert :string, :name
      convert :timestamp_to_datetime, :created_at
  # tables/20_Post.rb
  class Post < Converter
    @@old_table = "post"
    @@new_table = "posts"

    def run
      appended_FK :user_id, :user
      convert :int, :id
      convert :int, :user_id
      convert :string, :text
Here, the user table is converted in first place. The primary key of the old table, named `xy_user`, is `account_number`. It is
converted to `id`, the records with value 1 and 2 are skipped and those between 15 and 20 are appended as new record instead
of simply import them.

Then, the `post` table is proceeded. The `user_id` field refers to the id in `users`, so when the converter find a value between
15 and 20 ( which has been re-inserted ), it corrects it with the new id value. This is provided by the appended_FK() method.

For all the other records in `users` and for all the records in `posts` the converter check if the primary value exists in the new
table. If so, it updates the record ; else it inserts it.
Something went wrong with that request. Please try again.