Mongify allows you to map your data from a sql database and into a mongodb document database.
Ruby
Clone or download
Pull request Compare This branch is 2 commits ahead, 165 commits behind anlek:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
features
lib
spec
.gitignore
CHANGELOG.rdoc
Gemfile
Gemfile.lock
LICENSE
README.rdoc
Rakefile
mongify.gemspec

README.rdoc

Mongify

A data translator from sql database to mongoDB.

Supports MySQL, PostgreSQL, SQLite, Oracle, SQLServer, and DB2 (Basically anything ActiveRecord has built-in). However, I've only tested it with MySql and SQLite

Supports any version of MongoDB

Learn more about MongoDB at: www.mongodb.org

Install

gem install mongify

Usage

Creating a configuration file

In order for Mongify to do it's job, it needs to know where your databases are located.

Building a config file is as simple as this:

sql_connection do
  adaptor   "mysql"
  host      "localhost"
  username  "root"
  password  "passw0rd"
  database  "my_database"
end

mongodb_connection do
  host      "localhost"
  database  "my_database"
end

You can check your configuration by running

mongify check -c datbase.config

Options

Currently the only option that is support it is for the mongodb_connection.

mongodb_connection :force => true do                # Forcing a mongodb_connection will drop the database before processing
  # ...
end

You can omit the mongodb connection till you're ready to process your translation

Generating or create a translation

Generating a translation

If you're database is large and complex, it might be a bit too much to hand write the translation file, Mongify can help by running the translate command:

mongify translation -c database.config

Or pipe it right into a file by running

mongify translation -c database.config > translation_file.rb

Creating a translation

Creating a translation is pretty straight forward. It looks something like this:

table "users" do
  column "id", :key
  column "first_name", :string
  column "last_name", :string
  column "created_at", :datetime
  column "updated_at", :datetime
end

table "posts" do
  column "id", :key
  column "title", :string
  column "owner_id", :integer, :references => :users
  column "body", :text
  column "published_at", :datetime
  column "created_at", :datetime
  column "updated_at", :datetime
end

table "comments", :embed_in => :posts, :on => :post_id do
  column "id", :key
  column "body", :text
  column "post_id", :integer, :referneces => :posts
  column "user_id", :integer, :references => :users
  column "created_at", :datetime
  column "updated_at", :datetime
end

table "preferences", :embed_in => :users, :as => :object do
  column "id", :key
  column "user_id", :integer, :references => "users"
  column "notify_by_email", :boolean
end

table "notes", :embed_in => true, :polymorphic => 'notable' do
  column "id", :key
  column "user_id", :integer, :references => "users"
  column "notable_id", :integer
  column "notable_type", :string
  column "body", :text
  column "created_at", :datetime
  column "updated_at", :datetime
end

Save the file as "translation_file.rb" and run the command:

mongify process translation_file.rb -c database.config

Commands

Usage: mongify command [database_translation.rb] [-c database.config]

Commands:
  "check" or "ck"           >> Checks connection for sql and no_sql databases [configuration_file]
  "process" or "pr"         >> Takes a translation and process it to mongodb [configuration_file, translation_file]
  "translation" or "tr"     >> Outputs a translation file from a sql connection [configuration_file]

Examples:

  mongify translation -c datbase.config
  mongify tr -c database.config
  mongify check -c database.config
  mongify process database_translation.rb -c database.config

Common options:
    -h, --help                       Show this message
    -v, --version                    Show version
    -c, --config FILE                Configuration File to use

Translation Layout and Options

When dealing with a translation, there is a few options you can set

Table

Structure

Structure for defining a table is as follows:

table "table_name", {options} do
  # columns go here...
end

Options

Table Options are as follow:

table "table_name"                                        # Does a straight copy of the table
table "table_name", :embed_in => 'users'                  # Embeds table_name into users, assuming a user_id is present in table_name.
                                                          # This will also assume you want the table embedded as an array.

table "table_name",                                       # Embeds table_name into users, linking it via a owner_id
      :embed_in => 'users',                               # This will also assume you want the table embedded as an array.
      :on => 'owner_id'                                   

table "table_name",                                       # Embeds table_name into users as a one to one relationship
      :embed_in => 'users',                               # This also assumes you have a user_id present in table_name
      :on => 'owner_id',                                  # You can also specify both :on and :as options when embedding
      :as => 'object'                                     # NOTE: If you rename the owner_id column, make sure you 
                                                          # update the :on to the new column name

table "table_name", :rename_to => 'my_table'              # This will allow you to rename the table as it's getting process
                                                          # Just remember that columns that use :reference need to
                                                          # reference the new name.

table "table_name", :ignore => true                       # This will ignore the whole table (like it doesn't exist)
                                                          # This option is good for tables like: schema_migrations

table "table_name",                                       # This allows you to specify the table as being polymorphic
      :polymorphic => 'notable',                          # and provide the name of the polymorphic relationship.
      :embed_in => true                                   # Setting embed_in => true allows the relationship to be
                                                          # embedded directly into the parent class.
                                                          # If you do not embed it, the polymorphic table will be copied in to
                                                          # MongoDB and the notable_id will be updated to the new BSON::ObjectID

table "table_name" do                                     # A table can take a before_save block that will be called just
  before_save do |row|                                    # before the row is saved to the no sql database.
    row.admin = row.delete('permission').to_i > 50        # This gives you the ability to do very powerful things like:
  end                                                     # Moving records around, renaming records, changing values in row based on
end                                                       # some values! Checkout Mongify::Database::DataRow to learn more

More documentation can be found at {Mongify::Database::Table}

Columns

Structure

Structure for defining a column is as follows:

column "name", :type, {options}

Columns with no type given will be set to :string

Notes

Leaving a column out when defining a table will result in a copy of the information (as a string).

Types

Before we cover the options, you need to know what types of columns are supported:

:key                  # Columns that are primary keys need to be marked as :key type
:integer              # Will be converted to a integer
:float                # Will be converted to a float
:decimal              # Will be converted to a string (due to MongoDB Ruby Drivers not supporting BigDecimal, read details in Mongify::Database::Column under Decimal Storage)
:string               # Will be converted to a string
:text                 # Will be converted to a string
:datetime             # Will be converted to a Time format (DateTime is not currently supported in the Mongo ruby driver)
:date                 # Will be converted to a Time format (Date is not currently supported in the Mongo ruby driver)
:timestamps           # Will be converted to a Time format
:time                 # Will be converted to a Time format (the date portion of the Time object will be 2000-01-01)
:binary               # Will be converted to a string
:boolean              # Will be converted to a true or false values

Options

column "post_id", :integer, :referneces => :posts   # Referenced columns need to be marked as such, this will mean that they will be updated
                                                    # with the new BSON::ObjectID.
                                                    # NOTE: if you rename the table 'posts', you should set the :references to the new name

column "name", :string, :ignore => true             # Ignoring a column will make the column NOT copy over to the new database

column "surname", 
       :string, 
       :rename_to => 'last_name'                    # Rename_to allows you to rename the column                                                      

<em>For decimal columns you can specify a few options:</em>
column "total",                                     # This is a default conversion setting.
       :decimal,
       :as => 'string'                              

column "total",                                     # You can specify to convert your decimal to integer
        :decimal,                                   # specifying scale will define how many decimal places to keep
        :as => 'integer',                           # Example: :scale => 2 will convert 123.4567 to 12346 before saving
        :scale => 2

More documentation can be found at {Mongify::Database::Column}

Notes

I have currently only tested this in Ruby 1.8.7-p330

More testing will be done once the basic functionality is done

TODO

  • Allow deeper embedding

  • Test in Ruby 1.9

  • Test in different databases

Known Issues

  • Can't do anything to an embedded table

Development

Requirements

You just need bundler 1.0.8

gem install bundler
bundle install
copy over spec/support/database.example to spec/support/database.yml and fill in required info
rake test:mysql:setup

Special Thanks

Just want to thank my wife (Alessia) who not only puts up with me working on my free time but sometimes helps out listening to my craziness or helping me name classes or functions.

I would also like to thank Mon_Ouie on the Ruby IRC channel for helping me figure out how to setup the internal configuration file reading.

About

This gem was made by Andrew Kalek

Reach me at:

License

Copyright © 2011 Andrew Kalek, Anlek Consulting

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.