response.json()?

[pawelmhm]

python-requests have response.json() that decodes json body and returns appropriate Python objects. Does it make sense to have something like this in Scrapy?

[kmike]

@eliasdorneles proposed it here: #1729 (comment). I think that's fine to add it. Not sure if we need JsonResponse - maybe it'd be nice to have, to be able to check if json is available. But maybe it doesn't worths it, I'm not sure.

[kmike]

The question is whether should response.json raise AttributeError if response is not JSON; if it should (like .text, .css and .xpath) then we need JsonResponse. If it shouldn't then we don't need JsonResponse.

[pawelmhm]

python-requests provide it by default and just fail with ValueError if body is not json. Of course their implementation is not necessarily ideal, maybe we can do better. We could check for response headers and check if they have application/json content type, but there's a risk that some responses won't have json content type and they will be json (it's probably not standard compliant but I imagine it might be possible). It would also make code somewhat more complicated, e.g.:

# some spider callback
if isinstance(response, scrapy.response.JSONResponse):
    data = response.json()
else:
   raise ValueError('expected JSON')

instead of:

data = response.json()

[eliasdorneles]

@kmike oops, only .text is raising AttributeError: https://github.com/scrapy/scrapy/blob/master/scrapy/http/response/__init__.py#L86-L103 -- should I change that for the case people do hasattr(response, 'css') as well?

[eliasdorneles]

About response.json, I'd avoid creating a new subclass (JsonResponse) if we don't have a clear use for it.

I've been thinking that the .json() method could be on the selectors as well, as a shortcut to extract() -> json.loads, but that idea has to mature a little bit. :)

[IAlwaysBeCoding]

If we do include the response.json then could we also include a method for JmesQuery. Something like this response.jmes('[::].urls').extract(). Although, I don't think we will need extract_first() and just a simple extract() will do.

[kmike]

Regarding jmespath - see also scrapy/parsel#27.

As for .json(), is it just a shortcut for json.loads(response.text)?

Supporting .json() it in selectors sounds good, but I'd wait for real use cases / examples for that.

[twinsant]

mark

[ejulio]

I was thinking about .json and .jmespath this week.
I think that something like response.as_json() or response.css('script::text').as_json() would be nice, since sometimes we're handling JSON data and apis.

However, I don't think it should be only a wrapper for json.loads(response.text). Otherwise we would end up writing some code like

data = response.json()
item['field'] = data['property']['property']

But, it would be better to load the JSON as a python object and then write

data = response.json()
item['field'] = data.property.property

[tianhuil]

Would love to have this feature.

[JochenFromm]

I have added a PR which is being reviewed by @elacuesta at the moment, please let me know if it goes in the right direction.