Crashing on Non-UTF8 Character Encoding #39

Closed
tadman opened this Issue Nov 17, 2011 · 2 comments

Comments

2 participants

tadman commented Nov 17, 2011

I found a situation where the engine will crash hard on a specific kind of input. The result looks something like this:

ruby(48546,0x7fff7003cca0) malloc: *** error for object 0x101929e50: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Since Rails uses Psych as the default serializer now, any data with irregular characters could trigger this kind of problem. Most parts of Ruby will throw an "invalid UTF-8" exception instead of exploding like this.

I've created a minimal test-case that blows up on Ruby 1.9.2-p290 on OS X (64-bit):

#!/usr/bin/env ruby
# encoding: BINARY

# MacRoman encoded text "naïve"
naive = "na\x95ve"

require 'psych'

Psych.dump([ 'test', naive ])

Using rdebug I was able to extract a stack trace showing how this trip-up occurred:

/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:21:in `scalar'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:21:in `visit_Psych_Nodes_Scalar'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:6:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:26:in `block in visit_Psych_Nodes_Sequence'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:26:in `each'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:26:in `visit_Psych_Nodes_Sequence'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:8:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:16:in `block in visit_Psych_Nodes_Document'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:16:in `each'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:16:in `visit_Psych_Nodes_Document'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:10:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:10:in `block in visit_Psych_Nodes_Stream'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:10:in `each'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:10:in `visit_Psych_Nodes_Stream'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:11:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/nodes/node.rb:36:in `to_yaml'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych.rb:166:in `dump'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/base.rb:1720:in `block in arel_attributes_values'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/base.rb:1713:in `each'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/base.rb:1713:in `arel_attributes_values'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/persistence.rb:265:in `create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/timestamp.rb:47:in `create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:277:in `block in create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.0.7/lib/active_support/callbacks.rb:414:in `_run_create_callbacks'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:277:in `create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/persistence.rb:246:in `create_or_update'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:273:in `block in create_or_update'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.0.7/lib/active_support/callbacks.rb:419:in `_run_save_callbacks'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:273:in `create_or_update'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/persistence.rb:39:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/validations.rb:43:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/attribute_methods/dirty.rb:21:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:240:in `block (2 levels) in save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:292:in `block in with_transaction_returning_status'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:207:in `transaction'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:290:in `with_transaction_returning_status'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:240:in `block in save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:251:in `rollback_active_record_state!'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:239:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:273:in `block in create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:503:in `block in create_record'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:480:in `add_record_to_target_with_callbacks'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:503:in `create_record'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:271:in `create'
Owner

tenderlove commented Nov 17, 2011

I can't seem to reproduce this. Can you try with the latest psych gem? Make sure to do gem "psych" before requiring it.

tadman commented Nov 17, 2011

You're right in that it doesn't happen on the current release of psych, 1.2.2 from what I can tell, but you need to declare the gem dependency or it won't happen.

If ActiveRecord listed psych as a dependency this would've been avoided because, presumably, it would have fetched the most recent version. The one bundled with Ruby 1.9.2-p290 is, sadly, broken. It has to be manually added in the Gemfile to fix this.

Ruby 1.9.3-p0 is bundled with Psych 1.2.1 which produces a slightly different error, but at least no hard crash:

/opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:24:in `scalar': expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS (RuntimeError)
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:24:in `visit_Psych_Nodes_Scalar'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:15:in `visit'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:5:in `accept'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:29:in `block in visit_Psych_Nodes_Sequence'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:29:in `each'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:29:in `visit_Psych_Nodes_Sequence'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:15:in `visit'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:5:in `accept'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:19:in `block in visit_Psych_Nodes_Document'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:19:in `each'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:19:in `visit_Psych_Nodes_Document'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:15:in `visit'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:5:in `accept'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:13:in `block in visit_Psych_Nodes_Stream'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:13:in `each'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/emitter.rb:13:in `visit_Psych_Nodes_Stream'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:15:in `visit'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/visitors/visitor.rb:5:in `accept'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych/nodes/node.rb:46:in `to_yaml'
    from /opt/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/psych.rb:190:in `dump'
    from naive.rb:11:in `<main>'

The 1.2.2 version is fine in 1.9.3, though.

Thanks for the prompt reply. It would be great if the next build of 1.9.3 will have 1.2.2 or better bundled with it as this is probably going to hit more than a few unsuspecting people.

@tadman tadman closed this Nov 17, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment