New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON renderer uses invalid Content-Type #1611

Closed
dstufft opened this Issue Mar 17, 2015 · 5 comments

Comments

Projects
None yet
4 participants
@dstufft
Contributor

dstufft commented Mar 17, 2015

When using the JSON renderer Pyramid adds this header: Content-Type: application/json; charset=UTF-8. However according to the IANA the application/json media type does not actually support a charset. It is always in UTF-8 there is no other valid encoding.

@mmerickel

This comment has been minimized.

Show comment
Hide comment
@mmerickel

mmerickel Mar 17, 2015

Member

Note: No "charset" parameter is defined for this registration.
Adding one really has no effect on compliant recipients.

I guess I don't see what the issue is. The JSON RFC defines the content as UTF-8 by default but that other encodings are also valid. We are just being cautious, and according to your link still compliant, no?

Member

mmerickel commented Mar 17, 2015

Note: No "charset" parameter is defined for this registration.
Adding one really has no effect on compliant recipients.

I guess I don't see what the issue is. The JSON RFC defines the content as UTF-8 by default but that other encodings are also valid. We are just being cautious, and according to your link still compliant, no?

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 17, 2015

Contributor

It's not a major issue, but it's not compliant no. Any compliant parser will ignore the value because the application/json mimetype does not have any parameters (required or optional). It's as meaningless as putting application/json; frob=lob.

The only valid encodings for JSON is UTF8, UTF16, and UTF32. The way a compliant JSON parser determines encoding is by looking at the first four octects:

Encoding

JSON text SHALL be encoded in Unicode. The default encoding is
UTF-8.

Since the first two characters of a JSON text will always be ASCII
characters [RFC0020], it is possible to determine whether an octet
stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
at the pattern of nulls in the first four octets.

      00 00 00 xx  UTF-32BE
      00 xx 00 xx  UTF-16BE
      xx 00 00 00  UTF-32LE
      xx 00 xx 00  UTF-16LE
      xx xx xx xx  UTF-8

and

IANA Considerations

The MIME media type for JSON text is application/json.

Type name: application

Subtype name: json

Required parameters: n/a

Optional parameters: n/a

Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

JSON may be represented using UTF-8, UTF-16, or UTF-32. When JSON
is written in UTF-8, JSON is 8bit compatible. When JSON is
written in UTF-16 or UTF-32, the binary content-transfer-encoding
must be used.

FWIW I'm OK if this is wontfixed too, I just noticed that it was generating a meaningless (to a compliant parser) value and figured I'd open an issue incase y'all cared.

Contributor

dstufft commented Mar 17, 2015

It's not a major issue, but it's not compliant no. Any compliant parser will ignore the value because the application/json mimetype does not have any parameters (required or optional). It's as meaningless as putting application/json; frob=lob.

The only valid encodings for JSON is UTF8, UTF16, and UTF32. The way a compliant JSON parser determines encoding is by looking at the first four octects:

Encoding

JSON text SHALL be encoded in Unicode. The default encoding is
UTF-8.

Since the first two characters of a JSON text will always be ASCII
characters [RFC0020], it is possible to determine whether an octet
stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
at the pattern of nulls in the first four octets.

      00 00 00 xx  UTF-32BE
      00 xx 00 xx  UTF-16BE
      xx 00 00 00  UTF-32LE
      xx 00 xx 00  UTF-16LE
      xx xx xx xx  UTF-8

and

IANA Considerations

The MIME media type for JSON text is application/json.

Type name: application

Subtype name: json

Required parameters: n/a

Optional parameters: n/a

Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

JSON may be represented using UTF-8, UTF-16, or UTF-32. When JSON
is written in UTF-8, JSON is 8bit compatible. When JSON is
written in UTF-16 or UTF-32, the binary content-transfer-encoding
must be used.

FWIW I'm OK if this is wontfixed too, I just noticed that it was generating a meaningless (to a compliant parser) value and figured I'd open an issue incase y'all cared.

@mmerickel

This comment has been minimized.

Show comment
Hide comment
@mmerickel

mmerickel Mar 17, 2015

Member

Someone disagrees with you. The relevant code[1] is in webob (where this issue belongs). Webob is explicitly adding a charset to application/json. A simple fix if webob stays the same is to delete the charset after setting the content type.

resp = Response(content_type='application/json')
del resp.charset

resp = Response()
resp.content_type = 'application/json'
del resp.charset

resp = Response()
resp.headers['Content-Type'] = 'application/json'
# it looks like you can trick webob since it doesn't monitor headers for changes.

[1] https://github.com/Pylons/webob/blob/master/webob/response.py#L124

Member

mmerickel commented Mar 17, 2015

Someone disagrees with you. The relevant code[1] is in webob (where this issue belongs). Webob is explicitly adding a charset to application/json. A simple fix if webob stays the same is to delete the charset after setting the content type.

resp = Response(content_type='application/json')
del resp.charset

resp = Response()
resp.content_type = 'application/json'
del resp.charset

resp = Response()
resp.headers['Content-Type'] = 'application/json'
# it looks like you can trick webob since it doesn't monitor headers for changes.

[1] https://github.com/Pylons/webob/blob/master/webob/response.py#L124

@nnja

This comment has been minimized.

Show comment
Hide comment
@nnja

nnja Apr 14, 2015

Contributor

punt to webob

Contributor

nnja commented Apr 14, 2015

punt to webob

@mcdonc

This comment has been minimized.

Show comment
Hide comment
@mcdonc

mcdonc Jun 6, 2015

Member

Closing in this tracker, as it's now being tracked in WebOb.

Member

mcdonc commented Jun 6, 2015

Closing in this tracker, as it's now being tracked in WebOb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment