JSONP renderer fails to escape U+2028 and U+2029

[vfaronov]

When Unicode is enabled for JSON output, the JSONP renderer fails to escape U+2028 and U+2029 which are valid in JSON but invalid in JavaScript.

Try adding this simple model to the example app from the README:

from django.db import models

class Post(models.Model):

    title = models.CharField(max_length=100)
    content = models.TextField()

Create an object like this:

from example.models import Post
Post.objects.create(title=u'Test\u2028', content=u'example')

Then fire up a dev server and make an HTML page with a JSONP request like this:

<script type="text/javascript"
        src="http://127.0.0.1:8000/posts/?format=jsonp"></script>

Open the page in your browser and observe the error console. In Chromium, I get:

Uncaught SyntaxError: Unexpected token ILLEGAL

[tomchristie]

Which version of python is this btw?

[tomchristie]

Should this additionally be added as a Python bug.

Appears to be resolved in simplejson - https://github.com/simplejson/simplejson/blob/master/simplejson/encoder.py#L36
But not in CPython - https://hg.python.org/cpython/file/e1afae23dfcd/Lib/json/encoder.py#l26

And I don't see it listed as an issue?... http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=2028+2029+json&submit=search&status=-1%2C1%2C2%2C3

[vfaronov]

I run Python 2.7.3. This may be a nice thing to have in the standard json, but it’s not a bug, because what json.dumps() returns is perfectly valid as JSON, it’s only problematic when you try to interpret it as JavaScript.

[tomchristie]

So unsure whether to resolve this in JSONRenderer (as an enhancement) or in JSONPRenderer (as a bugfix).

Aside: Personally I do still class this as a bug in the Python json lib, as there's no good reason why it should just always be escaped. The JSON spec clearly has a bug when it refers to being derived from "a small subset of ECMAScript" http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf Unclear why they don't at least add an ammendment recommending that implementations escape those two characters.

[kevin-brown]

While I agree that it's an oversight in the JSON specification, I'm leaning towards it being an issue in JSONPRenderer for now considering it is valid JSON.

It may be useful to open a ticket to see if the CPython json module can also escape it by default. Escaping was added to simplejson in simplejson/simplejson@4989e69. The rails ticket for this is rails/rails#10320, and the same issue exists in the json gem.

[tomchristie]

It's resolved in rails, it's resolved in simplejson, not super impressed that the python ticket is classed as 'not a bug'. Disagree with that assessment.

[xordoquy]

Maybe we could make simplejson easy to use instead of the default json.

[tomchristie]

It's an option, although not strictly necessary to the resolution of this ticket, and would mean users have one extra thing to think about (rather than we just deal with it quietly)