Permalink
Browse files

Add per-database type translation support for schema changes, transla…

…ting ruby classes to database specific types

After many requests for per-database type translation, and my
repeatedly saying I'll accept a good patch, I went ahead and coded
the support myself.  Here's what the API looks like:

    DB.create_table(:cats) do
      String :a
      Integer :b
      Fixnum :c
      Bignum :d
      Float :e
      BigDecimal :f
      Date :g
      DateTime :h
      Time :i
      Numeric :j
      File :k
      TrueClass :l
      FalseClass :m
      column :n, Fixnum
      primary_key :o, :type=>String
      foreign_key :p, :f, :type=>Date
    end

Basically, anywhere you would use a symbol or string to specify a
type, you use the ruby class itself that you are trying to represent.
There is support in the MySQL, SQLite, and PostgreSQL adapters for
these mappings, and the defaults should be reasonable, with most
other databases probably only needing to override a few types.

This first form of column description was already accepted, so this
breaks compatibility.  In generally, most schema modification methods
would be specified in all lowercase, so this is not likely to affect
many people.  This also breaks compatibility by changing the meaning
of #Float, #Integer, and #String inside Sequel::SQL::Generator, since
they previously were Kernel private instance methods.   One of the
reasons I'm not as worried about compatibility is that since Float,
Integer, and String were already defined, using the above syntax
would not have worked well before, because String :a wouldn't create
a column (it would return "a"), and Integer :b and Float :e would
have raised errors.

This commit includes a big expansion to the integration type tests,
testing all of the generic types that Sequel supports.  Most
databases don't pass these yet.  SQLite fails on the File test
because it can't support "\0" values in strings.  PostgreSQL and
MySQL pass on ruby and ruby 1.9, but generally fail at least one test
when using JDBC or DataObjects.  The H2 JDBC subadapter fails on a
Timestamp column for reasons I don't quite understand, and also fails
the embedded "\0" values in strings test.

This commit updates the native SQLite adapter to convert float, real,
and double precision columns to Floats.

This commit adds support for conversion of Java::JavaSQL::Date to
Date and Java::JavaMath::BigDecimal to BigDecimal in the JDBC
adapter.
  • Loading branch information...
jeremyevans committed Jan 27, 2009
1 parent 4b57b05 commit 9efc8ae695077f75a7298a6fc8134fba041aa988
View
@@ -1,5 +1,7 @@
=== HEAD
+* Add per-database type translation support for schema changes, translating ruby classes to database specific types (jeremyevans)
+
* Add Sequel::DatabaseConnectionError, for indicating that Sequel wasn't able to connect to the database (jeremyevans)
* Add validates_not_string validation, useful in conjunction with raise_on_typecast_failure = false (jeremyevans)
@@ -421,10 +421,14 @@ def convert_type(v)
case v
when Java::JavaSQL::Timestamp, Java::JavaSQL::Time
v.to_string.to_sequel_time
+ when Java::JavaSQL::Date
+ v.to_string.to_date
when Java::JavaIo::BufferedReader
lines = []
while(line = v.read_line) do lines << line end
lines.join("\n")
+ when Java::JavaMath::BigDecimal
+ BigDecimal.new(v.to_string)
else
v
end
@@ -16,7 +16,8 @@ module DatabaseMethods
NOT_NULL = Sequel::Schema::SQL::NOT_NULL
NULL = Sequel::Schema::SQL::NULL
PRIMARY_KEY = Sequel::Schema::SQL::PRIMARY_KEY
- TYPES = Sequel::Schema::SQL::TYPES
+ TYPES = Sequel::Schema::SQL::TYPES.merge(DateTime=>'datetime', \
+ TrueClass=>'tinyint', FalseClass=>'tinyint')
UNIQUE = Sequel::Schema::SQL::UNIQUE
UNSIGNED = Sequel::Schema::SQL::UNSIGNED
@@ -123,6 +124,11 @@ def schema_parse_table(table_name, opts)
[row.delete(:Field).to_sym, row]
end
end
+
+ # Override the standard type conversions with MySQL specific ones
+ def type_literal_base(column)
+ TYPES[column[:type]]
+ end
end
# Dataset methods shared by datasets that use MySQL databases.
@@ -216,6 +222,8 @@ def literal(v)
BOOL_TRUE
when false
BOOL_FALSE
+ when DateTime, Time
+ v.strftime("'%Y-%m-%d %H:%M:%S'")
else
super
end
@@ -167,6 +167,7 @@ module DatabaseMethods
SQL_ROLLBACK = 'ROLLBACK'.freeze
SQL_RELEASE_SAVEPOINT = 'RELEASE SAVEPOINT autopoint_%d'.freeze
SYSTEM_TABLE_REGEXP = /^pg|sql/.freeze
+ TYPES = Sequel::Schema::SQL::TYPES.merge(File=>'bytea')
# Creates the function in the database. See create_function_sql for arguments.
def create_function(*args)
@@ -550,6 +551,11 @@ def schema_parser_dataset(table_name, opts)
def sql_function_args(args)
"(#{Array(args).map{|a| Array(a).reverse.join(' ')}.join(', ')})"
end
+
+ # Override the standard type conversions with PostgreSQL specific ones
+ def type_literal_base(column)
+ TYPES[column[:type]]
+ end
end
# Instance methods for datasets that connect to a PostgreSQL database.
@@ -5,6 +5,7 @@ module DatabaseMethods
SYNCHRONOUS = {'0' => :off, '1' => :normal, '2' => :full}.freeze
TABLES_FILTER = "type = 'table' AND NOT name = 'sqlite_sequence'"
TEMP_STORE = {'0' => :default, '1' => :file, '2' => :memory}.freeze
+ TYPES = Sequel::Schema::SQL::TYPES.merge(Bignum=>'integer')
# Run all alter_table commands in a transaction. This is technically only
# needed for drop column.
@@ -95,14 +96,6 @@ def identifier_output_method_default
nil
end
- # SQLite supports schema parsing using the table_info PRAGMA, so
- # parse the output of that into the format Sequel expects.
- def schema_parse_table(table_name, opts)
- parse_pragma(table_name, opts).map do |row|
- [row.delete(:name).to_sym, row]
- end
- end
-
def parse_pragma(table_name, opts)
self["PRAGMA table_info(?)", table_name].map do |row|
row.delete(:cid)
@@ -115,6 +108,19 @@ def parse_pragma(table_name, opts)
row
end
end
+
+ # SQLite supports schema parsing using the table_info PRAGMA, so
+ # parse the output of that into the format Sequel expects.
+ def schema_parse_table(table_name, opts)
+ parse_pragma(table_name, opts).map do |row|
+ [row.delete(:name).to_sym, row]
+ end
+ end
+
+ # Override the standard type conversions with SQLite specific ones
+ def type_literal_base(column)
+ TYPES[column[:type]]
+ end
end
# Instance methods for datasets that connect to an SQLite database
@@ -46,6 +46,12 @@ def connect(server)
db.translator.add_translator("decimal", &prok)
db.translator.add_translator("money", &prok)
+ # Handle floating point values with Float
+ prok = proc{|t,v| Float(v) rescue v}
+ db.translator.add_translator("float", &prok)
+ db.translator.add_translator("real", &prok)
+ db.translator.add_translator("double precision", &prok)
+
db
end
@@ -424,7 +424,7 @@ def select(*args)
# Default serial primary key options.
def serial_primary_key_options
- {:primary_key => true, :type => :integer, :auto_increment => true}
+ {:primary_key => true, :type => Integer, :auto_increment => true}
end
# Returns true if the database is using a single-threaded connection pool.
@@ -13,6 +13,10 @@ module Schema
# allowing users to specify column type as a method instead of using
# the column method, which makes for a nicer DSL.
class Generator
+ # Classes specifying generic types that Sequel will convert to database-specific types.
+ GENERIC_TYPES=[String, Integer, Fixnum, Bignum, Float, Numeric, BigDecimal,
+ Date, DateTime, Time, File, TrueClass, FalseClass]
+
# Set the database in which to create the table, and evaluate the block
# in the context of this object.
def initialize(db, &block)
@@ -23,6 +27,16 @@ def initialize(db, &block)
instance_eval(&block) if block
end
+ # Add a method for each of the given types that creates a column
+ # with that type as a constant. Types given should either already
+ # be constants/classes or a capitalized string/symbol with the same name
+ # as a constant/class.
+ def self.add_type_method(*types)
+ types.each do |type|
+ class_eval "def #{type}(name, opts={}); column(name, #{type}, opts); end"
+ end
+ end
+
# Add a unnamed constraint to the DDL, specified by the given block
# or args.
def check(*args, &block)
@@ -87,10 +101,10 @@ def foreign_key(name, table=nil, opts = {})
when NilClass
opts
else
- raise(Error, "The seconds argument to foreign_key should be a Hash, Symbol, or nil")
+ raise(Error, "The second argument to foreign_key should be a Hash, Symbol, or nil")
end
return composite_foreign_key(name, opts) if name.is_a?(Array)
- column(name, :integer, opts)
+ column(name, Integer, opts)
end
# Add a full text index on the given columns to the DDL.
@@ -173,6 +187,8 @@ def composite_foreign_key(columns, opts)
@columns << {:type => :check, :constraint_type => :foreign_key,
:name => nil, :columns => columns }.merge(opts)
end
+
+ add_type_method(*GENERIC_TYPES)
end
# Schema::AlterTableGenerator is an internal class that the user is not expected
@@ -222,7 +238,7 @@ def add_unique_constraint(columns, opts = {})
# use the composite key syntax even if it is only one column.
def add_foreign_key(name, table, opts = {})
return add_composite_foreign_key(name, table, opts) if name.is_a?(Array)
- add_column(name, :integer, {:table=>table}.merge(opts))
+ add_column(name, Integer, {:table=>table}.merge(opts))
end
# Add a full text index on the given columns to the DDL for the table.
@@ -12,7 +12,11 @@ module SQL
SET_DEFAULT = 'SET DEFAULT'.freeze
SET_NULL = 'SET NULL'.freeze
TYPES = Hash.new {|h, k| k}
- TYPES[:double] = 'double precision'
+ TYPES.merge!(:double=>'double precision', String=>'varchar',
+ Integer=>'integer', Fixnum=>'integer', Bignum=>'bigint',
+ Float=>'double precision', BigDecimal=>'numeric', Numeric=>'numeric',
+ Date=>'date', DateTime=>'timestamp', Time=>'timestamp', File=>'blob',
+ TrueClass=>'boolean', FalseClass=>'boolean')
UNDERSCORE = '_'.freeze
UNIQUE = ' UNIQUE'.freeze
UNSIGNED = ' UNSIGNED'.freeze
@@ -312,9 +316,10 @@ def schema_column_type(db_type)
# SQL fragment specifying the type of a given column.
def type_literal(column)
- column[:size] ||= 255 if column[:type] == :varchar
+ type = type_literal_base(column)
+ column[:size] ||= 255 if type.to_s == 'varchar'
elements = column[:size] || column[:elements]
- "#{type_literal_base(column)}#{literal(Array(elements)) if elements}#{UNSIGNED if column[:unsigned]}"
+ "#{type}#{literal(Array(elements)) if elements}#{UNSIGNED if column[:unsigned]}"
end
# SQL fragment specifying the base type of a given column,
@@ -1,43 +1,86 @@
require File.join(File.dirname(__FILE__), 'spec_helper.rb')
describe "Supported types" do
- def create_items_table_with_column(name, type)
- INTEGRATION_DB.create_table!(:items){column name, type}
+ def create_items_table_with_column(name, type, opts={})
+ INTEGRATION_DB.create_table!(:items){column name, type, opts}
INTEGRATION_DB[:items]
end
specify "should support NULL correctly" do
- ds = create_items_table_with_column(:number, :integer)
+ ds = create_items_table_with_column(:number, Integer)
ds.insert(:number => nil)
ds.all.should == [{:number=>nil}]
end
- specify "should support integer type" do
- ds = create_items_table_with_column(:number, :integer)
+ specify "should support generic integer type" do
+ ds = create_items_table_with_column(:number, Integer)
ds.insert(:number => 2)
ds.all.should == [{:number=>2}]
end
+
+ specify "should support generic fixnum type" do
+ ds = create_items_table_with_column(:number, Fixnum)
+ ds.insert(:number => 2)
+ ds.all.should == [{:number=>2}]
+ end
+
+ specify "should support generic bignum type" do
+ ds = create_items_table_with_column(:number, Bignum)
+ ds.insert(:number => 2**34)
+ ds.all.should == [{:number=>2**34}]
+ end
+
+ specify "should support generic float type" do
+ ds = create_items_table_with_column(:number, Float)
+ ds.insert(:number => 2.1)
+ ds.all.should == [{:number=>2.1}]
+ end
+
+ specify "should support generic numeric type" do
+ ds = create_items_table_with_column(:number, Numeric, :size=>[15, 10])
+ ds.insert(:number => BigDecimal.new('2.123456789'))
+ ds.all.should == [{:number=>BigDecimal.new('2.123456789')}]
+ ds = create_items_table_with_column(:number, BigDecimal, :size=>[15, 10])
+ ds.insert(:number => BigDecimal.new('2.123456789'))
+ ds.all.should == [{:number=>BigDecimal.new('2.123456789')}]
+ end
- specify "should support varchar type" do
- ds = create_items_table_with_column(:name, 'varchar(255)'.lit)
+ specify "should support generic string type" do
+ ds = create_items_table_with_column(:name, String)
ds.insert(:name => 'Test User')
ds.all.should == [{:name=>'Test User'}]
end
- specify "should support date type" do
- ds = create_items_table_with_column(:dat, :date)
+ specify "should support generic date type" do
+ ds = create_items_table_with_column(:dat, Date)
d = Date.today
ds.insert(:dat => d)
- x = ds.first[:dat]
- x = x.iso8601.to_date if Time === x
- x.to_s.should == d.to_s
+ ds.first[:dat].should == d
end
- specify "should support time type" do
- ds = create_items_table_with_column(:tim, :time)
+ specify "should support generic datetime type" do
+ ds = create_items_table_with_column(:tim, DateTime)
+ t = DateTime.now
+ ds.insert(:tim => t)
+ ds.first[:tim].strftime('%Y%m%d%H%M%S').should == t.strftime('%Y%m%d%H%M%S')
+ ds = create_items_table_with_column(:tim, Time)
t = Time.now
ds.insert(:tim => t)
- x = ds.first[:tim]
- [t.strftime('%H:%M:%S'), t.iso8601].should include(x.respond_to?(:strftime) ? x.strftime('%H:%M:%S') : x.to_s)
+ ds.first[:tim].strftime('%Y%m%d%H%M%S').should == t.strftime('%Y%m%d%H%M%S')
+ end
+
+ specify "should support generic file type" do
+ ds = create_items_table_with_column(:name, File)
+ ds.insert(:name => ("a\0"*300).to_blob)
+ ds.all.should == [{:name=>("a\0"*300).to_blob}]
+ end
+
+ specify "should support generic boolean type" do
+ ds = create_items_table_with_column(:number, TrueClass)
+ ds.insert(:number => true)
+ ds.all.should == [{:number=>true}]
+ ds = create_items_table_with_column(:number, FalseClass)
+ ds.insert(:number => true)
+ ds.all.should == [{:number=>true}]
end
end
@@ -47,9 +47,9 @@
it "creates foreign key column" do
@columns[3][:name].should == :parent_id
- @columns[3][:type].should == :integer
+ @columns[3][:type].should == Integer
@columns[6][:name].should == :node_id
- @columns[6][:type].should == :integer
+ @columns[6][:type].should == Integer
end
it "uses table for foreign key columns, if specified" do
@@ -133,10 +133,35 @@
{:op => :add_constraint, :type => :check, :constraint_type => :check, :name => :con1, :check => ['fred > 100']},
{:op => :drop_constraint, :name => :con2},
{:op => :add_constraint, :type => :check, :constraint_type => :unique, :name => :con3, :columns => [:aaa, :bbb, :ccc]},
- {:op => :add_column, :name => :id, :type => :integer, :primary_key=>true, :auto_increment=>true},
- {:op => :add_column, :name => :node_id, :type => :integer, :table=>:nodes},
+ {:op => :add_column, :name => :id, :type => Integer, :primary_key=>true, :auto_increment=>true},
+ {:op => :add_column, :name => :node_id, :type => Integer, :table=>:nodes},
{:op => :add_constraint, :type => :check, :constraint_type => :primary_key, :columns => [:aaa, :bbb]},
{:op => :add_constraint, :type => :check, :constraint_type => :foreign_key, :columns => [:node_id, :prop_id], :table => :nodes_props}
]
end
end
+
+describe "Sequel::Schema::Generator generic type methods" do
+ before do
+ @generator = Sequel::Schema::Generator.new(SchemaDummyDatabase.new) do
+ String :a
+ Integer :b
+ Fixnum :c
+ Bignum :d
+ Float :e
+ BigDecimal :f
+ Date :g
+ DateTime :h
+ Time :i
+ Numeric :j
+ File :k
+ TrueClass :l
+ FalseClass :m
+ end
+ @columns, @indexes = @generator.create_info
+ end
+
+ it "should store the type class in :type for each column" do
+ @columns.map{|c| c[:type]}.should == [String, Integer, Fixnum, Bignum, Float, BigDecimal, Date, DateTime, Time, Numeric, File, TrueClass, FalseClass]
+ end
+end
Oops, something went wrong.

0 comments on commit 9efc8ae

Please sign in to comment.