Generator does not actually set encoding #137

Open
headius opened this Issue May 31, 2012 · 8 comments

3 participants

@headius

Here's the reduced test case I came up with, from a larger case by @rkh:

# encoding: utf-8
require 'json'
x = 'foó'
p x.to_json
p x.to_json.encoding

The resulting json string ends up encoded as ASCII-8BIT. My probably-wrong patch is:

diff --git a/java/src/json/ext/Generator.java b/java/src/json/ext/Generator.java
index ecceb27..6ea5aa3 100644
--- a/java/src/json/ext/Generator.java
+++ b/java/src/json/ext/Generator.java
@@ -391,6 +391,7 @@ public final class Generator {
                 }

                 session.getStringEncoder().encode(src.getByteList(), buffer);
+                buffer.setEncoding(src.getByteList().getEncoding());
             }
         };

I doubt this is right, but I could not find anywhere that json lib is actually setting the final encoding of the target.

@flori
Owner

I cannot reproduce this problem:

jruby-1.6.7.2 :090 > require 'json'
 => true
jruby-1.6.7.2 :010 > JSON.generator
 => JSON::Ext::Generator 
jruby-1.6.7.2 :011 > JSON::VERSION
 => "1.7.3" 
jruby-1.6.7.2 :012 > RUBY_DESCRIPTION
 => "jruby 1.6.7.2 (ruby-1.9.2-p312) (2012-05-01 26e08ba) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_31) [darwin-x86_64-java]" 
jruby-1.6.7.2 :013 > x = 'foó'
 => "foó" 
jruby-1.6.7.2 :014 > x.encoding
 => #<Encoding:UTF-8> 
jruby-1.6.7.2 :015 > x.to_json
 => "\"foó\"" 
jruby-1.6.7.2 :016 > x.to_json.encoding
 => #<Encoding:UTF-8>

The encoding is set in the method

private static abstract class Handler<T extends IRubyObject>

before the new string is returned.

@rkh
rkh commented Jun 7, 2012
git clone gh
cd gh
bundle install
bundle exec rspec ./spec/response_spec.rb:5 
@rkh
rkh commented Jun 7, 2012

Or, using your example:

>> x = 'foó'
=> "foó" 
>> x.encoding
=> #<Encoding:UTF-8> 
>> JSON.load(x.to_json).encoding
=> #<Encoding:ASCII-8BIT>
@flori
Owner

Ah, that's interesting. It could be the load, does parse work instead?

@rkh
rkh commented Jun 7, 2012

Nope, same result.

@flori
Owner

Ok, I am trying to reproduce this again when I am at a real Computer.

@flori
Owner

I tried "my" example from above and it works here:

(flori@silverblade:gh [master] ☯@gh ->0)$ jruby --1.9 -S irb
jruby-1.6.7.2 :001 > require 'json'
 => true 
jruby-1.6.7.2 :002 > x = 'foó'
 => "foó" 
jruby-1.6.7.2 :003 > x.encoding
 => #<Encoding:UTF-8> 
jruby-1.6.7.2 :004 > JSON.load(x.to_json).encoding
 => #<Encoding:UTF-8> 
jruby-1.6.7.2 :005 > JSON::VERSION
 => "1.7.3" 
jruby-1.6.7.2 :006 > RUBY_DESCRIPTION
 => "jruby 1.6.7.2 (ruby-1.9.2-p312) (2012-05-01 26e08ba) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_31) [darwin-x86_64-java]" 

Are we using the same JSON and jruby versions?

@flori
Owner

So, when I run your spec an exception is raised in JRuby, isn't this rather #138 than #137?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment