Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Remove unicode character encoding from ActiveSupport::JSON.encode

The encoding scheme (e.g. ☠ -> "\u2620") was broken for characters
not in the Basic Multilingual Plane.  It is possible to escape them
for json using the weird encoding scheme of a twelve-character
sequence representing the UTF-16 surrogate pair (e.g. '𠜎' ->
"\u270e\u263a") but this wasn't properly handled in the escaping code.
Since raw UTF-8 is allowed in json, it was decided to simply pass
through the raw bytes rather than attempt to escape them.

Backport of zbskii@9ace3a8

Conflicts:

	activesupport/CHANGELOG.md
	activesupport/lib/active_support/json/encoding.rb
	activesupport/test/json/encoding_test.rb
  • Loading branch information...
commit 815a9431ab61376a7e8e1bdff21f87bc557992f8 1 parent 9c581d3
zbskii zbskii authored steveklabnik committed
7 activesupport/CHANGELOG.md
View
@@ -1,5 +1,12 @@
## Rails 3.2.10 (unreleased)
+* Remove surrogate unicode character encoding from ActiveSupport::JSON.encode
+ The encoding scheme was broken for unicode characters outside the basic
+ multilingual plane; since json is assumed to be UTF-8, and we already force the
+ encoding to UTF-8 simply pass through the un-encoded characters.
+
+ *Brett Carter*
+
* Fix mocha v0.13.0 compatibility. *James Mead*
* `#as_json` isolates options when encoding a hash. [Backport #8185]
8 activesupport/lib/active_support/json/encoding.rb
View
@@ -122,13 +122,7 @@ def escape(string)
if string.respond_to?(:force_encoding)
string = string.encode(::Encoding::UTF_8, :undef => :replace).force_encoding(::Encoding::BINARY)
end
- json = string.
- gsub(escape_regex) { |s| ESCAPED_CHARS[s] }.
- gsub(/([\xC0-\xDF][\x80-\xBF]|
- [\xE0-\xEF][\x80-\xBF]{2}|
- [\xF0-\xF7][\x80-\xBF]{3})+/nx) { |s|
- s.unpack("U*").pack("n*").unpack("H*")[0].gsub(/.{4}/n, '\\\\u\&')
- }
+ json = string.gsub(escape_regex) { |s| ESCAPED_CHARS[s] }
json = %("#{json}")
json.force_encoding(::Encoding::UTF_8) if json.respond_to?(:force_encoding)
json
19 activesupport/test/json/encoding_test.rb
View
@@ -100,11 +100,11 @@ def test_hash_encoding
def test_utf8_string_encoded_properly_when_kcode_is_utf8
with_kcode 'UTF8' do
result = ActiveSupport::JSON.encode('€2.99')
- assert_equal '"\\u20ac2.99"', result
+ assert_equal '"€2.99"', result
assert_equal(Encoding::UTF_8, result.encoding) if result.respond_to?(:encoding)
result = ActiveSupport::JSON.encode('✎☺')
- assert_equal '"\\u270e\\u263a"', result
+ assert_equal '"✎☺"', result
assert_equal(Encoding::UTF_8, result.encoding) if result.respond_to?(:encoding)
end
end
@@ -113,11 +113,24 @@ def test_utf8_string_encoded_properly_when_kcode_is_utf8
def test_non_utf8_string_transcodes
s = ''.encode('Shift_JIS')
result = ActiveSupport::JSON.encode(s)
- assert_equal '"\\u4e8c"', result
+ assert_equal '""', result
assert_equal Encoding::UTF_8, result.encoding
end
end
+ def test_wide_utf8_chars
+ w = '𠜎'
+ result = ActiveSupport::JSON.encode(w)
+ assert_equal '"𠜎"', result
+ end
+
+ def test_wide_utf8_roundtrip
+ hash = { string: "𐒑" }
+ json = ActiveSupport::JSON.encode(hash)
+ decoded_hash = ActiveSupport::JSON.decode(json)
+ assert_equal "𐒑", decoded_hash['string']
+ end
+
def test_exception_raised_when_encoding_circular_reference_in_array
a = [1]
a << a
Please sign in to comment.
Something went wrong with that request. Please try again.