Permalink
Browse files

Force binary data inserted for a string data type to utf-8 and log an

error.  Strings tagged as binary will be stored in sqlite3 as blobs.  It
is an error to insert binary data to a string column, so an error is
emitted in the log file.  People are highly encouraged to track down the
source of the binary strings and make sure that the encoding is set
correctly before inserting to the database.
  • Loading branch information...
1 parent 635b9d4 commit 42a7979cf1810c75343b18007858b81718b90678 @tenderlove tenderlove committed Aug 25, 2011
@@ -161,10 +161,25 @@ def quoted_date(value) #:nodoc:
end
end
- def type_cast(value, column) # :nodoc:
- return super unless BigDecimal === value
+ if "<3".encoding_aware?
@dmathieu
dmathieu Aug 26, 2011 Contributor

❤️ too

@spastorino
spastorino Aug 26, 2011 Member

Why is not encoding_aware? a class method?

@tenderlove
tenderlove Aug 26, 2011 Member

Because strings know about encodings, not classes. :-P

+ def type_cast(value, column) # :nodoc:
+ return value.to_f if BigDecimal === value
+ return super unless String === value
+ return super unless column && value
+
+ value = super
+ if column.type == :string && value.encoding == Encoding::ASCII_8BIT
+ @logger.error "Binary data inserted for `string` type on column `#{column.name}`"
+ value.encode! 'utf-8'
+ end
+ value
+ end
+ else
+ def type_cast(value, column) # :nodoc:
+ return super unless BigDecimal === value
- value.to_f
+ value.to_f
+ end
end
# DATABASE STATEMENTS ======================================
@@ -1,5 +1,6 @@
# encoding: utf-8
require "cases/helper"
+require 'models/owner'
module ActiveRecord
module ConnectionAdapters
@@ -19,6 +20,21 @@ def setup
eosql
end
+ def test_column_types
+ return skip('only test encoding on 1.9') unless "<3".encoding_aware?
+
+ owner = Owner.create!(:name => "hello".encode('ascii-8bit'))
+ owner.reload
+ select = Owner.columns.map { |c| "typeof(#{c.name})" }.join ', '
+ result = Owner.connection.exec_query <<-esql
+ SELECT #{select}
+ FROM #{Owner.table_name}
+ WHERE #{Owner.primary_key} = #{owner.id}
+ esql
+
+ assert(!result.rows.first.include?("blob"), "should not store blobs")
+ end
+
def test_exec_insert
column = @conn.columns('items').find { |col| col.name == 'number' }
vals = [[column, 10]]

4 comments on commit 42a7979

@sikachu
Member

Finally! So this would fix the issue I discussed with you about text encoding right?

@tenderlove
Member

Yes, but it will emit an error to your log file. You should set the encoding, but this will let you know when you aren't.

@bsodmike
Contributor

This is causing a failure on Travis:

Finished tests in 69.537812s, 45.1984 tests/s, 138.3276 assertions/s.

1) Failure:
test_column_types(ActiveRecord::ConnectionAdapters::SQLite3AdapterTest)
[/home/vagrant/builds/rails/rails/activerecord/test/cases/adapters/sqlite3/sqlite3_adapter_test.rb:35]:
should not store blobs

3143 tests, 9619 assertions, 1 failures, 0 errors, 0 skips
@guilleiguaran
Member

The failure was caused by a bug in Ruby 1.9.3: http://redmine.ruby-lang.org/issues/5287, it's fixed already. I have tested with 1.9.3 (head instead of preview1) and the test is passing now.

Please sign in to comment.