-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Upgrade yaml_db to latest version (11678745cb1)
- Loading branch information
1 parent
03e5e20
commit cbb2106
Showing
17 changed files
with
739 additions
and
353 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# YamlDb | ||
|
||
YamlDb is a database-independent format for dumping and restoring data. It complements the the database-independent schema format found in db/schema.rb. The data is saved into db/data.yml. | ||
|
||
This can be used as a replacement for mysqldump or pg_dump, but only for the databases typically used by Rails apps. Users, permissions, schemas, triggers, and other advanced database features are not supported - by design. | ||
|
||
Any database that has an ActiveRecord adapter should work. | ||
|
||
## Installation | ||
|
||
This plugin can now be installed as a gem via | ||
|
||
gem install yaml_db | ||
|
||
Note that when doing this, the rake tasks won't automatically be available to your app. You'll have to explicitly include them via the approaches followed: [here](http://ggr.com/how-to-include-a-gems-rake-tasks-in-your-rails-app.html), [here](https://rails.lighthouseapp.com/projects/8994/tickets/510-rake-tasks-not-included-for-gems), [and here](https://rails.lighthouseapp.com/projects/8994/tickets/59) | ||
|
||
## Usage | ||
|
||
rake db:data:dump -> Dump contents of Rails database to db/data.yml | ||
rake db:data:load -> Load contents of db/data.yml into the database | ||
|
||
Further, there are tasks db:dump and db:load which do the entire database (the equivalent of running db:schema:dump followed by db:data:load). Also, there are other tasks recently added that allow the export of the database contents to/from multiple files (each one named after the table being dumped or loaded). | ||
|
||
rake db:data:dump_dir -> Dump contents of database to curr_dir_name/tablename.extension (defaults to yaml) | ||
rake db:data:load_dir -> Load contents of db/data_dir into database | ||
|
||
In addition, we have plugins whereby you can export your database to/from various formats. We only deal with yaml and csv right now, but you can easily write tools for your own formats (such as Excel or XML). To use another format, just load setting the "class" parameter to the class you are using. This defaults to "YamlDb::Helper" which is a refactoring of the old yaml_db code. We'll shorten this to use class nicknames in a little bit. | ||
|
||
## Examples | ||
|
||
One common use would be to switch your data from one database backend to another. For example, let's say you wanted to switch from SQLite to MySQL. You might execute the following steps: | ||
|
||
1. rake db:dump | ||
|
||
2. Edit config/database.yml and change your adapter to mysql, set up database params | ||
|
||
3. mysqladmin create [database name] | ||
|
||
4. rake db:load | ||
|
||
## Credits | ||
|
||
Created by Orion Henry and Adam Wiggins. Major updates by Ricardo Chimal, Jr. | ||
|
||
Patches contributed by Michael Irwin, Tom Locke, and Tim Galeckas. | ||
|
||
Send questions, feedback, or patches to the Heroku mailing list: http://groups.google.com/group/heroku | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
0.1.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
#require 'FasterCSV' | ||
module CsvDb | ||
module Helper | ||
def self.loader | ||
Load | ||
end | ||
|
||
def self.dumper | ||
Dump | ||
end | ||
|
||
def self.extension | ||
"csv" | ||
end | ||
end | ||
|
||
class Load < SerializationHelper::Load | ||
def self.load_documents(io, truncate = true) | ||
tables = {} | ||
curr_table = nil | ||
io.each do |line| | ||
if /BEGIN_CSV_TABLE_DECLARATION(.+)END_CSV_TABLE_DECLARATION/ =~ line | ||
curr_table = $1 | ||
tables[curr_table] = {} | ||
else | ||
if tables[curr_table]["columns"] | ||
tables[curr_table]["records"] << FasterCSV.parse(line)[0] | ||
else | ||
tables[curr_table]["columns"] = FasterCSV.parse(line)[0] | ||
tables[curr_table]["records"] = [] | ||
end | ||
end | ||
end | ||
|
||
tables.each_pair do |table_name, contents| | ||
load_table(table_name, contents, truncate) | ||
end | ||
end | ||
end | ||
|
||
class Dump < SerializationHelper::Dump | ||
|
||
def self.before_table(io,table) | ||
io.write "BEGIN_CSV_TABLE_DECLARATION#{table}END_CSV_TABLE_DECLARATION\n" | ||
end | ||
|
||
def self.dump(io) | ||
tables.each do |table| | ||
before_table(io, table) | ||
dump_table(io, table) | ||
after_table(io, table) | ||
end | ||
end | ||
|
||
def self.after_table(io,table) | ||
io.write "" | ||
end | ||
|
||
def self.dump_table_columns(io, table) | ||
io.write(table_column_names(table).to_csv) | ||
end | ||
|
||
def self.dump_table_records(io, table) | ||
|
||
column_names = table_column_names(table) | ||
|
||
each_table_page(table) do |records| | ||
rows = SerializationHelper::Utils.unhash_records(records, column_names) | ||
records.each do |record| | ||
io.write(record.to_csv) | ||
end | ||
end | ||
end | ||
|
||
end | ||
|
||
|
||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,195 @@ | ||
module SerializationHelper | ||
|
||
class Base | ||
attr_reader :extension | ||
|
||
def initialize(helper) | ||
@dumper = helper.dumper | ||
@loader = helper.loader | ||
@extension = helper.extension | ||
end | ||
|
||
def dump(filename) | ||
disable_logger | ||
@dumper.dump(File.new(filename, "w")) | ||
reenable_logger | ||
end | ||
|
||
def dump_to_dir(dirname) | ||
Dir.mkdir(dirname) | ||
tables = @dumper.tables | ||
tables.each do |table| | ||
io = File.new "#{dirname}/#{table}.#{@extension}", "w" | ||
@dumper.before_table(io, table) | ||
@dumper.dump_table io, table | ||
@dumper.after_table(io, table) | ||
end | ||
end | ||
|
||
def load(filename, truncate = true) | ||
disable_logger | ||
@loader.load(File.new(filename, "r"), truncate) | ||
reenable_logger | ||
end | ||
|
||
def load_from_dir(dirname, truncate = true) | ||
Dir.entries(dirname).each do |filename| | ||
if filename =~ /^[.]/ | ||
next | ||
end | ||
@loader.load(File.new("#{dirname}/#{filename}", "r"), truncate) | ||
end | ||
end | ||
|
||
def disable_logger | ||
@@old_logger = ActiveRecord::Base.logger | ||
ActiveRecord::Base.logger = nil | ||
end | ||
|
||
def reenable_logger | ||
ActiveRecord::Base.logger = @@old_logger | ||
end | ||
end | ||
|
||
class Load | ||
def self.load(io, truncate = true) | ||
ActiveRecord::Base.connection.transaction do | ||
load_documents(io, truncate) | ||
end | ||
end | ||
|
||
def self.truncate_table(table) | ||
begin | ||
ActiveRecord::Base.connection.execute("TRUNCATE #{SerializationHelper::Utils.quote_table(table)}") | ||
rescue Exception | ||
ActiveRecord::Base.connection.execute("DELETE FROM #{SerializationHelper::Utils.quote_table(table)}") | ||
end | ||
end | ||
|
||
def self.load_table(table, data, truncate = true) | ||
column_names = data['columns'] | ||
if truncate | ||
truncate_table(table) | ||
end | ||
load_records(table, column_names, data['records']) | ||
reset_pk_sequence!(table) | ||
end | ||
|
||
def self.load_records(table, column_names, records) | ||
if column_names.nil? | ||
return | ||
end | ||
columns = column_names.map{|cn| ActiveRecord::Base.connection.columns(table).detect{|c| c.name == cn}} | ||
quoted_column_names = column_names.map { |column| ActiveRecord::Base.connection.quote_column_name(column) }.join(',') | ||
quoted_table_name = SerializationHelper::Utils.quote_table(table) | ||
records.each do |record| | ||
quoted_values = record.zip(columns).map{|c| ActiveRecord::Base.connection.quote(c.first, c.last)}.join(',') | ||
ActiveRecord::Base.connection.execute("INSERT INTO #{quoted_table_name} (#{quoted_column_names}) VALUES (#{quoted_values})") | ||
end | ||
end | ||
|
||
def self.reset_pk_sequence!(table_name) | ||
if ActiveRecord::Base.connection.respond_to?(:reset_pk_sequence!) | ||
ActiveRecord::Base.connection.reset_pk_sequence!(table_name) | ||
end | ||
end | ||
|
||
|
||
end | ||
|
||
module Utils | ||
|
||
def self.unhash(hash, keys) | ||
keys.map { |key| hash[key] } | ||
end | ||
|
||
def self.unhash_records(records, keys) | ||
records.each_with_index do |record, index| | ||
records[index] = unhash(record, keys) | ||
end | ||
|
||
records | ||
end | ||
|
||
def self.convert_booleans(records, columns) | ||
records.each do |record| | ||
columns.each do |column| | ||
next if is_boolean(record[column]) | ||
record[column] = (record[column] == 't' or record[column] == '1') | ||
end | ||
end | ||
records | ||
end | ||
|
||
def self.boolean_columns(table) | ||
columns = ActiveRecord::Base.connection.columns(table).reject { |c| silence_warnings { c.type != :boolean } } | ||
columns.map { |c| c.name } | ||
end | ||
|
||
def self.is_boolean(value) | ||
value.kind_of?(TrueClass) or value.kind_of?(FalseClass) | ||
end | ||
|
||
def self.quote_table(table) | ||
ActiveRecord::Base.connection.quote_table_name(table) | ||
end | ||
|
||
end | ||
|
||
class Dump | ||
def self.before_table(io, table) | ||
|
||
end | ||
|
||
def self.dump(io) | ||
tables.each do |table| | ||
before_table(io, table) | ||
dump_table(io, table) | ||
after_table(io, table) | ||
end | ||
end | ||
|
||
def self.after_table(io, table) | ||
|
||
end | ||
|
||
def self.tables | ||
ActiveRecord::Base.connection.tables.reject { |table| ['schema_info', 'schema_migrations'].include?(table) } | ||
end | ||
|
||
def self.dump_table(io, table) | ||
return if table_record_count(table).zero? | ||
|
||
dump_table_columns(io, table) | ||
dump_table_records(io, table) | ||
end | ||
|
||
def self.table_column_names(table) | ||
ActiveRecord::Base.connection.columns(table).map { |c| c.name } | ||
end | ||
|
||
|
||
def self.each_table_page(table, records_per_page=1000) | ||
total_count = table_record_count(table) | ||
pages = (total_count.to_f / records_per_page).ceil - 1 | ||
id = table_column_names(table).first | ||
boolean_columns = SerializationHelper::Utils.boolean_columns(table) | ||
quoted_table_name = SerializationHelper::Utils.quote_table(table) | ||
|
||
(0..pages).to_a.each do |page| | ||
sql = ActiveRecord::Base.connection.add_limit_offset!("SELECT * FROM #{quoted_table_name} ORDER BY #{id}", | ||
:limit => records_per_page, :offset => records_per_page * page | ||
) | ||
records = ActiveRecord::Base.connection.select_all(sql) | ||
records = SerializationHelper::Utils.convert_booleans(records, boolean_columns) | ||
yield records | ||
end | ||
end | ||
|
||
def self.table_record_count(table) | ||
ActiveRecord::Base.connection.select_one("SELECT COUNT(*) FROM #{SerializationHelper::Utils.quote_table(table)}").values.first.to_i | ||
end | ||
|
||
end | ||
|
||
end |
Oops, something went wrong.