Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

rails 3.2.13 breaks/unexpectedly changes json encoding of some utf8 characters #9819

Closed
RepublicOfAppsLLC opened this Issue Mar 20, 2013 · 7 comments

Comments

Projects
None yet
5 participants

The json encoding of some legal utf8 characters has changed between rails 3.2.12 and 3.2.13. I didn't see anything about this in the release notes, and it breaks some api integrations. For example, consider the json output

# Rails 3.2.12
input "polished look – instantly"
produces json "polished look \u2013 instantly"

# Rails 3.2.13
input "polished look – instantly"
produces json "polished look – instantly"

This change is unexpected, and produces issues when diffing output, since the input has not changed, but rails 3.2.13 has changed the output.

I see similar problems with é (\u00e9).

If you run the output through ActiveSupport::JSON.decode then the output compares equal, i.e.,

# Rubyish pseudocode
ActiveSupport::JSON.decode("polished look \u2013 instantly") == ActiveSupport::JSON.decode("polished look – instantly")

However, there is no guarantee that the output of a Rails json api is being consumed in Ruby. Nor is there any guarantee that it is being parsed by a json decoder before use. Somebody might be comparing output of curl or similar using file diffs. I know that the order of keys in json maps is not guaranteed, but diffing json api output before/after an upgrade is not uncommon, and it's common to do so with simple tools like curl and diff.

I'm just giving you a friendly heads up that you might get other reports of this. Thanks for all of your hard work in getting rails 3.2.13 out the door.

I consider this a bug/regression/unexpected behavior, and I think others will as well.

Thanks!

Member

steveklabnik commented Mar 20, 2013

There is another ticket about this, but yes, apparently we're now returning a UTF-8 string where before it was US-ASCII.

Contributor

jstorimer commented Mar 20, 2013

@steveklabnik can you link the other ticket? Seeing something similar.

Member

steveklabnik commented Mar 20, 2013

I don't remember it offhand, and I'm about to step away from the computer or I'd search.

Contributor

jstorimer commented Mar 20, 2013

I'll look harder, search didn't show an obvious match.

Contributor

timhaines commented Mar 20, 2013

I'm striking a similar problem that I haven't nailed down yet. I thought it was database related, but this could be it.

My solution was do this:

# Before 3.2.13
value = ... a json string scraped from my rails api ...
file.write value

# After 3.2.13
value = ... the same json string scraped from my rails api ...
file.write value.force_encoding "UTF-8"

Doing that seemed to get rid of the crazy "\xE2" from ASCII-8BIT to UTF-8 exceptions that I was seeing after upgrading to Rails 3.2.13. I also had to do my comparisons as

# This works in 3.2.13
ActiveSupport::JSON.decode(value1) == ActiveSupport::JSON.decode(value2)

# This used to work in 3.2.12, but is broken in 3.2.13
value1 == value2

Just posting this in case it helps anybody ...

Owner

rafaelfranca commented Mar 20, 2013

Duplication of #9498

@mallowlabs mallowlabs referenced this issue in codefirst/AsakusaSatellite Apr 10, 2013

Closed

Rollback to Rails 3.2.12 #36

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment