Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Strings are stored as BLOBs using ruby-1.9.3-preview1 #45

Closed
julian7 opened this Issue Aug 23, 2011 · 6 comments

Comments

Projects
None yet
2 participants

julian7 commented Aug 23, 2011

@gucki reported a strange behavior of factory_rails using 1.9.3-preview1 (thoughtbot/factory_girl#166), which was (mis)identified as a ruby error.

However, the problem is more general. It happens with AR timestamp fields too, as my tests showed (see my comment there for details).

While I could prepare a failing test for this issue, I went the other route: I bisected ruby from v1_9_2_290 to v1_9_3_preview1, and the first error occured at ruby/ruby@a8f5a06.

I believe while the change itself was in ruby, sqlite3-ruby should be fixed: my tests using 1.9.3-preview1 + mysql2 were succeeded.

Owner

tenderlove commented Aug 24, 2011

Strings tagged as ASCII-8BIT will be stored as binary. This is so that sqlite3 can distinguish between binary blobs, and actual text. (For example storing files File.binread("xxx")).

The problem you're experiencing indicates that some value is not being properly tagged with the correct encoding. Here is a demo program:

require 'sqlite3'

class Inserter
  def initialize
    @db = SQLite3::Database.new ':memory:'
    @db.execute "create table foo (name varchar(255))"
    @stmt = @db.prepare "insert into foo (name) values (?)"
  end

  def insert str
    @stmt.reset!
    @stmt.execute str
  end

  def inspect
    @db.execute "select name, typeof(name) from foo"
  end
end


o = Inserter.new

[
  'hello',
  'world'.encode('ascii-8bit'),
  Time.now.to_s,
].each do |string|
  o.insert string
  p o
end

__END__
Output:

[["hello", "text"]]
[["hello", "text"], ["world", "blob"]]
[["hello", "text"], ["world", "blob"], ["2011-08-24 15:59:01 -0700", "blob"]]

I think this is a problem in Ruby itself (and has been since 1.9.2), so I filed a ticket with patch here.

Make sure the strings you're putting in the database are tagged with the correct encoding, and sqlite3 will not store them as blobs.

@tenderlove tenderlove closed this Aug 24, 2011

julian7 commented Aug 25, 2011

Awesome! There must be two other issues in factory_girl and fabrication too :)

Owner

tenderlove commented Aug 25, 2011

@julian7 I've added a workaround to rails that will emit an error in your logs. That should fix your problems, but it will log an error if binary data is inserted in a string column. Really the best thing to do is find the source of the binary data and make sure it's tagged correctly.

Owner

tenderlove commented Aug 25, 2011

This workaround will ship with Rails 3.1.1.

Owner

tenderlove commented Aug 25, 2011

The Time#to_s problem will be fixed in Ruby 1.9.4.

julian7 commented Dec 16, 2011

This issue's afterlife is just amazing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment