Getting incompatible character encodings error on ruby 1.9.2 w/ protocol buffers

[myronmarston]

I just tried switching to protocol buffers now that 0.9 supports it, and I'm getting lots of test failures related to character encoding incompatibilities. Here's the pertinent part of the backtrace:

 Encoding::CompatibilityError:
   incompatible character encodings: US-ASCII and ASCII-8BIT
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake/buffer/base.rb:80:in `<<'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake/buffer/encode.rb:109:in `append_string'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake/buffer/encode.rb:13:in `append'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake.rb:91:in `block in encode!'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake.rb:87:in `each'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake.rb:87:in `encode!'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake.rb:79:in `block in encode'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake.rb:72:in `each'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/beefcake-0.2.1/lib/beefcake.rb:72:in `encode'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/riak-client-0.9.0/lib/riak/client/beefcake_protobuffs_backend.rb:133:in `write_protobuff'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/riak-client-0.9.0/lib/riak/client/beefcake_protobuffs_backend.rb:68:in `store_object'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/riak-client-0.9.0/lib/riak/robject.rb:141:in `store'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/document/persistence.rb:79:in `really_save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/callbacks.rb:57:in `block (2 levels) in really_save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.5/lib/active_support/callbacks.rb:418:in `_run_create_callbacks'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.5/lib/active_support/callbacks.rb:93:in `run_callbacks'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/callbacks.rb:56:in `block in really_save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.5/lib/active_support/callbacks.rb:423:in `_run_save_callbacks'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/activesupport-3.0.5/lib/active_support/callbacks.rb:93:in `run_callbacks'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/callbacks.rb:55:in `really_save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/document/persistence.rb:73:in `save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/attribute_methods/dirty.rb:24:in `save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/validations.rb:66:in `save'
 # /Users/myron/.rvm/gems/ruby-1.9.2-p180@silo/gems/ripple-0.9.0/lib/ripple/validations.rb:72:in `save!'

I'm not doing anything special to set encodings on strings anywhere, so I think it's likely that this is a bug in ripple (or beefcake).

Thanks,
Myron

[lucaspiller]

Hi Myron,

I believe it is a gems issue. I had this originally, but then recreated my Gemset and all the tests pass now:

$ ruby -v                                                                                                                                                                                                                               
ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-darwin10.6.0]
$ gem list                                                                                                                                                                        1 ↵
*** LOCAL GEMS ***
activemodel (3.0.5)
activesupport (3.0.5)
beefcake (0.2.1)
builder (2.1.2)
bundler (1.0.10)
capistrano (2.5.19)
capistrano-ext (1.2.1)
crack (0.1.8)
curb (0.7.10)
diff-lcs (1.1.2)
eventmachine (0.12.10)
excon (0.5.6)
highline (1.6.1)
httparty (0.7.4)
httpclient (2.1.7.2, 2.1.6.1)
i18n (0.5.0)
net-scp (1.0.4)
net-sftp (2.0.5)
net-ssh (2.1.0)
net-ssh-gateway (1.0.1)
nokogiri (1.4.4)
rack (1.2.1)
rake (0.8.7)
riak-client (0.9.0, 0.8.3)
ripple (0.9.0, 0.8.3)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
sinatra (1.1.3)
tilt (1.2.2)
whois (2.0.1)
yajl-ruby (0.8.2, 0.8.1)

Regards,

Luca Spiller

[myronmarston]

@lucaspiller: Thanks. I reinstalled all my gems and am getting the same error, unfortunately.

Did you install the exact same versions of all gems? Or was there a particular gem you identified as the culprit?

Myron

[seancribbs]

@myronmarston: I'd like to consider this bug critical, to be patched on 0.9-stable. I ran into this bug this morning and was unable to resolve it, even when using force_encoding all over the pbufs code. Unfortunately beefcake doesn't give much indication about which field was being serialized when the encoding error occurs.

[rb2k]

I also have this problem on beefcake 0.3.1 (or 0.2.1 for that matter) on ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-darwin10.6.0] with ripple 0.9

[seancribbs]

This should be fixed on 0.9-stable and master. Please comment/reopen if you encounter the problem again.

[charl]

I am still seeing this when using ':protocol => "pbc"':

Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8

Here's my setup:

$ ruby -v
jruby 1.7.2 (1.9.3p327) 2013-01-04 302c706 on OpenJDK 64-Bit Server VM 1.7.0_09-b30 [linux-amd64]

$ gem list

*** LOCAL GEMS ***

beefcake (0.3.7)
bouncy-castle-java (1.5.0146.1)
builder (3.1.4)
bundler (1.2.3)
celluloid (0.12.4)
excon (0.16.10)
facter (1.6.17)
i18n (0.6.1)
innertube (1.0.2)
jruby-launcher (1.0.15 java)
jruby-openssl (0.8.2)
json (1.7.6 java)
multi_json (1.5.0)
nokogiri (1.5.6 java)
rake (10.0.3)
redis (3.0.2)
rethinkdb (1.2.6.1)
riak-client (1.1.1)
ruby_protobuf (0.4.11)
rubygems-bundler (1.1.0)
rvm (1.11.3.6, 1.11.3.5)
spoon (0.0.1)
spoon_daemon (0.0.3)
timers (1.1.0)

Here's the stack trace:

Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8 -> 
org/jruby/RubyString.java:7563:in `encode', 
json/ext/GeneratorState.java:210:in `generate', 
/home/charl/.rvm/rubies/jruby-1.7.2/lib/ruby/1.9/json/common.rb:223:in `generate',
/home/charl/Projects/89n/analytics/thoth.git/lib/thoth/query.rb:282:in `get', 
/home/charl/Projects/89n/analytics/thoth.git/lib/thoth/query.rb:346:in `process', 
org/jruby/RubyKernel.java:1809:in `public_send', 
/home/charl/.rvm/gems/jruby-1.7.2@thoth/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in `dispatch', 
/home/charl/.rvm/gems/jruby-1.7.2@thoth/gems/celluloid-0.12.4/lib/celluloid/actor.rb:327:in `handle_message', 
/home/charl/.rvm/gems/jruby-1.7.2@thoth/gems/celluloid-0.12.4/lib/celluloid/tasks/task_fiber.rb:24:in `initialize', 
/home/charl/.rvm/gems/jruby-1.7.2@thoth/gems/celluloid-0.12.4/lib/celluloid/tasks/task_fiber.rb:23:in `initialize'

HTTP (with or without Excon) works perfectly.

[charl]

Hmm... this may not be a ripple specific issue. Should I rather move this to a ticket under https://github.com/basho/riak-ruby-client/issues or somewhere else?

[seancribbs]

@charl Yes, it would be most appropriate there.

[charl]

OK, created basho/riak-ruby-client#75.