Permalink
Browse files

Adding support for DataMapper

Basically we had to change every usage of

ActiveRecord::Base.connection
to use the new abstraction we created.

This led to more problems than expected when dealing with threading issues and connection availability to each thread.
  • Loading branch information...
filipesabella committed Jan 30, 2013
1 parent 798168b commit ba34c750090b98e00730d7abdf05d441a6c23238
View
@@ -8,3 +8,4 @@ gemfile:
- gemfiles/ar-2.3_mysql.gemfile
- gemfiles/ar-3.2_mysql.gemfile
- gemfiles/ar-3.2_mysql2.gemfile
+ - gemfiles/dm_mysql.gemfile
View
@@ -22,7 +22,7 @@ is great if you are using this engine, but only solves half the problem.
At SoundCloud we started having migration pains quite a while ago, and after
looking around for third party solutions, we decided to create our
own. We called it Large Hadron Migrator, and it is a gem for online
-ActiveRecord migrations.
+ActiveRecord and DataMapper migrations.
![LHC](http://farm4.static.flickr.com/3093/2844971993_17f2ddf2a8_z.jpg)
@@ -35,18 +35,22 @@ without locking the table. In contrast to [OAK][0] and the
[facebook tool][1], we only use a copy table and triggers.
The Large Hadron is a test driven Ruby solution which can easily be dropped
-into an ActiveRecord migration. It presumes a single auto incremented
-numerical primary key called id as per the Rails convention. Unlike the
-[twitter solution][2], it does not require the presence of an indexed
+into an ActiveRecord or DataMapper migration. It presumes a single auto
+incremented numerical primary key called id as per the Rails convention. Unlike
+the [twitter solution][2], it does not require the presence of an indexed
`updated_at` column.
## Requirements
Lhm currently only works with MySQL databases and requires an established
-ActiveRecord connection.
+ActiveRecord or DataMapper connection.
It is compatible and [continuously tested][4] with Ruby 1.8.7 and Ruby 1.9.x,
-ActiveRecord 2.3.x and 3.x as well as mysql and mysql2 adapters.
+ActiveRecord 2.3.x and 3.x (mysql and mysql2 adapters), as well as DataMapper
+1.2 (dm-mysql-adapter).
+
+Lhm also works with dm-master-slave-adapter, it'll bind to the master before
+running the migrations.
## Installation
@@ -66,6 +70,10 @@ ActiveRecord::Base.establish_connection(
:database => 'lhm'
)
+# or with DataMapper
+Lhm.setup(DataMapper.setup(:default, 'mysql://127.0.0.1/lhm'))
+
+# and migrate
Lhm.change_table :users do |m|
m.add_column :arbitrary, "INT(12)"
m.add_index [:arbitrary_id, :created_at]
@@ -97,7 +105,33 @@ class MigrateUsers < ActiveRecord::Migration
end
```
-**Note:** LHM won't delete the old, leftover table. This is on purpose, in order to prevent accidental data loss.
+Using dm-migrations, you'd define all your migrations as follows, and then call
+`migrate_up!` or `migrate_down!` as normal.
+
+```ruby
+require 'dm-migrations/migration_runner'
+require 'lhm'
+
+migration 1, :migrate_users do
+ up do
+ Lhm.change_table :users do |m|
+ m.add_column :arbitrary, "INT(12)"
+ m.add_index [:arbitrary_id, :created_at]
+ m.ddl("alter table %s add column flag tinyint(1)" % m.name)
+ end
+ end
+
+ down do
+ Lhm.change_table :users do |m|
+ m.remove_index [:arbitrary_id, :created_at]
+ m.remove_column :arbitrary
+ end
+ end
+end
+```
+
+**Note:** Lhm won't delete the old, leftover table. This is on purpose, in order
+to prevent accidental data loss.
## Table rename strategies
View
@@ -17,4 +17,3 @@ end
task :specs => [:unit, :integration]
task :default => :specs
-
View
@@ -6,15 +6,15 @@ set -u
source ~/.lhm
lhmkill() {
- ps -ef | gsed -n "/[m]ysqld.*lhm-cluster/p" | awk '{ print $2 }' | xargs kill
- sleep 5
+ echo killing lhm-cluster
+ ps -ef | sed -n "/[m]ysqld.*lhm-cluster/p" | awk '{ print $2 }' | xargs kill
+ sleep 2
}
echo stopping other running mysql instance
launchctl remove com.mysql.mysqld || { echo launchctl did not remove mysqld; }
"$mysqldir"/bin/mysqladmin shutdown || { echo mysqladmin did not shut down anything; }
-echo killing lhm-cluster
lhmkill
echo removing $basedir
@@ -0,0 +1,5 @@
+source :rubygems
+
+gem 'dm-core'
+gem 'dm-mysql-adapter'
+gemspec :path=>"../"
View
@@ -21,7 +21,5 @@ Gem::Specification.new do |s|
s.add_development_dependency "minitest", "= 2.10.0"
s.add_development_dependency "rake"
-
- s.add_dependency "activerecord"
end
View
@@ -1,9 +1,9 @@
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
# Schmidt
-require 'active_record'
require 'lhm/table'
require 'lhm/invoker'
+require 'lhm/connection'
require 'lhm/version'
# Large hadron migrator - online schema change tool
@@ -34,12 +34,25 @@ module Lhm
# @return [Boolean] Returns true if the migration finishes
# @raise [Error] Raises Lhm::Error in case of a error and aborts the migration
def self.change_table(table_name, options = {}, &block)
- connection = ActiveRecord::Base.connection
+ connection = Connection.new(adapter)
+
origin = Table.parse(table_name, connection)
invoker = Invoker.new(origin, connection)
block.call(invoker.migrator)
invoker.run(options)
true
end
+
+ def self.setup(adapter)
+ @@adapter = adapter
+ end
+
+ def self.adapter
+ @@adapter ||=
+ begin
+ raise 'Please call Lhm.setup' unless defined?(ActiveRecord)
+ ActiveRecord::Base.connection
+ end
+ end
end
@@ -13,7 +13,6 @@ module Lhm
# Lhm::SqlHelper.supports_atomic_switch?.
class AtomicSwitcher
include Command
- include SqlHelper
attr_reader :connection
@@ -36,14 +35,15 @@ def atomic_switch
end
def validate
- unless table?(@origin.name) && table?(@destination.name)
+ unless @connection.table_exists?(@origin.name) &&
+ @connection.table_exists?(@destination.name)
error "`#{ @origin.name }` and `#{ @destination.name }` must exist"
end
end
private
def execute
- sql statements
+ @connection.sql(statements)
end
end
end
View
@@ -83,7 +83,7 @@ def validate
def execute
up_to do |lowest, highest|
- affected_rows = update(copy(lowest, highest))
+ affected_rows = @connection.update(copy(lowest, highest))
if affected_rows > 0
sleep(throttle_seconds)
View
@@ -0,0 +1,143 @@
+module Lhm
+ require 'lhm/sql_helper'
+
+ class Connection
+ def self.new(adapter)
+ if defined?(DataMapper) && adapter.is_a?(DataMapper::Adapters::AbstractAdapter)
+ DataMapperConnection.new(adapter)
+ elsif defined?(ActiveRecord)
+ ActiveRecordConnection.new(adapter)
+ else
+ raise 'Neither DataMapper nor ActiveRecord found.'
+ end
+ end
+
+ class DataMapperConnection
+ include SqlHelper
+
+ def initialize(adapter)
+ @adapter = adapter
+ @database_name = adapter.options['path'][1..-1]
+ end
+
+ def sql(statements)
+ [statements].flatten.each do |statement|
+ execute(tagged(statement))
+ end
+ end
+
+ def show_create(table_name)
+ sql = "show create table `#{ table_name }`"
+ select_values(sql).last
+ end
+
+ def current_database
+ @database_name
+ end
+
+ def update(statements)
+ [statements].flatten.inject(0) do |memo, statement|
+ result = @adapter.execute(tagged(statement))
+ memo += result.affected_rows
+ end
+ end
+
+ def select_all(sql)
+ @adapter.select(sql).to_a
+ end
+
+ def select_one(sql)
+ select_all(sql).first
+ end
+
+ def select_values(sql)
+ select_one(sql).values
+ end
+
+ def select_value(sql)
+ select_one(sql)
+ end
+
+ def destination_create(origin)
+ original = %{CREATE TABLE "#{ origin.name }"}
+ replacement = %{CREATE TABLE "#{ origin.destination_name }"}
+
+ sql(origin.ddl.gsub(original, replacement))
+ end
+
+ def execute(sql)
+ @adapter.execute(sql)
+ end
+
+ def table_exists?(table_name)
+ !!select_one(%Q{
+ select *
+ from information_schema.tables
+ where table_schema = '#{ @database_name }'
+ and table_name = '#{ table_name }'
+ })
+ end
+ end
+
+ class ActiveRecordConnection
+ include SqlHelper
+
+ def initialize(adapter)
+ @adapter = adapter
+ @database_name = @adapter.current_database
+ end
+
+ def sql(statements)
+ [statements].flatten.each do |statement|
+ execute(tagged(statement))
+ end
+ end
+
+ def show_create(table_name)
+ sql = "show create table `#{ table_name }`"
+ specification = nil
+ execute(sql).each { |row| specification = row.last }
+ specification
+ end
+
+ def current_database
+ @database_name
+ end
+
+ def update(sql)
+ @adapter.update(sql)
+ end
+
+ def select_all(sql)
+ @adapter.select_all(sql)
+ end
+
+ def select_one(sql)
+ @adapter.select_one(sql)
+ end
+
+ def select_values(sql)
+ @adapter.select_values(sql)
+ end
+
+ def select_value(sql)
+ @adapter.select_value(sql)
+ end
+
+ def destination_create(origin)
+ original = %{CREATE TABLE `#{ origin.name }`}
+ replacement = %{CREATE TABLE `#{ origin.destination_name }`}
+
+ sql(origin.ddl.gsub(original, replacement))
+ end
+
+ def execute(sql)
+ @adapter.execute(sql)
+ end
+
+ def table_exists?(table_name)
+ @adapter.table_exists?(table_name)
+ end
+ end
+ end
+end
View
@@ -68,21 +68,21 @@ def trigger(type)
end
def validate
- unless table?(@origin.name)
+ unless @connection.table_exists?(@origin.name)
error("#{ @origin.name } does not exist")
end
- unless table?(@destination.name)
+ unless @connection.table_exists?(@destination.name)
error("#{ @destination.name } does not exist")
end
end
def before
- sql(entangle)
+ @connection.sql(entangle)
end
def after
- sql(untangle)
+ @connection.sql(untangle)
end
def revert
@@ -53,19 +53,20 @@ def uncommitted(&block)
end
def validate
- unless table?(@origin.name) && table?(@destination.name)
+ unless @connection.table_exists?(@origin.name) &&
+ @connection.table_exists?(@destination.name)
error "`#{ @origin.name }` and `#{ @destination.name }` must exist"
end
end
private
def revert
- sql "unlock tables"
+ @connection.sql("unlock tables")
end
def execute
- sql statements
+ @connection.sql(statements)
end
end
end
Oops, something went wrong.

0 comments on commit ba34c75

Please sign in to comment.