json request for certain media types fail

[iherman]

Long story short

The resp.json() call reports a bug if the return type is something like application/XXX+json (as opposed to application/json). However, application/XXX+json is a perfectly fine media type, "extending" the semantics (not the syntax) of the core json syntax, ie, should be parsed as bona fide json...

Your environment

OS X 10.11.6, Python 3.6.2

[jscholes]

You can pass a custom content_type parameter to the ClientResponse.json method, although I agree that this isn't perhaps the most sensible API. I just switched some code from using Requests to using this library and while Requests could decode the JSON without a problem, aiohttp reported an error which will trip some people up.

[asvetlov]

Related to #2174

[asvetlov]

Extending the list of accepted json content types is fair easy change.
Please propose a Pull Request. application/XXX+json is not acceptable but I pretty sure you could grab a list of wide spread json types from requests library.

[asvetlov]

use content_type=None for disabling the check

[iherman]

@asvetlov, unfortunately I missed the reply on Sep 11; probably my fault.

However, I am not sure I agree with your statement that "application/XXX+json is not acceptable". In my view, it is perfectly acceptable (and future proof) if the code simply ignores the XXX+ part, whatever XXX is. Looking at RFC 6839:

At the same time, using the suffix allows receivers of the media types to do generic processing of the underlying representation in cases where

• they do not need to perform special handling of the particular semantics of the exact media type, and

• there is no special knowledge needed by such a generic processor in order to parse that underlying representation other than what would be needed to parse any example of that underlying representation.

I.e., there is no reason for aiohttp to maintain whatever specific list. This is valid for any conversion step, not only for JSON.

---

Yes, I agree that using content_type=None can be done, but that is clearly just a hack...

[asvetlov]

What to do?
On one hand we want to prevent processing non-JSON data in .json() method.
On other we don't want to be too strict.

Maybe you can pass a custom content type when fetching a data from your server?
Or create a custom json reading function on top of resp.read()?

[iherman]

@asvetlov,

The fundamental fact is that data coming with XXX+json is JSON. From the point of view of aiohttp that all that matters. A properly registered XXX+json media type means that the content is JSON but it may have some additional syntactic or structural constraint within the framework of JSON. This may mean imposing a particular JSON Schema, or some more complicated specification on top of what JSON means, or may be no extra syntactic restriction whatsoever.

To take a specific example. application/ld+json is a bona fide media type defined through the structural syntax; the content is defined via the JSON-LD Recommendation. That specification adds some additional semantics to individual terms, requires some specific structure to the JSON data, but the essential point is that a JSON-LD content is a perfectly fine JSON content. (This is probably one of the most complex examples. Others may be a few paragraphs only.)

In my view, this means that the handling of the XXX+json (or XXX+xml, etc.) is simple, as far as aiohttp goes: strip the XXX+ string altogether (the + character is not allowed elsewhere in a media type name), and handle the content as json (or xml). You are done. All other processing is in the application domain.

(You will find an abundance of such media types on the media type registry.)

[asvetlov]

You would say that any application/xxx+json is a valid JSON type.
Or parsing all JSON registered types from the media registry is better?

[webknjaz]

I'd say both statements are true.