Fix: response.json() call makes unnecessary memory allocation

[jxlil]

Now response.json() uses response.body to retrieve the dict object, so the str response is no longer stored in RAM.

Closes #5968

[codecov]

Codecov Report

Merging #6016 (af7cad4) into master (3318971) will increase coverage by 0.01%.
Report is 3 commits behind head on master.
The diff coverage is 100.00%.

❗ Current head af7cad4 differs from pull request most recent head 4dd3267. Consider uploading reports for the commit 4dd3267 to get more accurate results

@@            Coverage Diff             @@
##           master    #6016      +/-   ##
==========================================
+ Coverage   88.85%   88.87%   +0.01%     
==========================================
  Files         162      162              
  Lines       11445    11445              
  Branches     1861     1861              
==========================================
+ Hits        10170    10172       +2     
  Misses        968      968              
+ Partials      307      305       -2     

Files Changed	Coverage Δ
scrapy/extensions/corestats.py	100.00% <100.00%> (ø)
scrapy/extensions/feedexport.py	95.05% <100.00%> (+2.32%)	⬆️
scrapy/http/response/text.py	98.42% <100.00%> (ø)

... and 2 files with indirect coverage changes

[GeorgeA92]

From my side It looks OK.

However I am still not sure about.. additional tests that need (or not need) to implement here (I will prefer to wait for reviews from other contributors).

[kmike]

The behavior is not exactly the same, if the response is not UTF-8, and properly indicates it.

From https://datatracker.ietf.org/doc/html/rfc7159#section-8.1:

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32.

From https://docs.python.org/3/library/json.html#json.loads:

s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.

So we should be fine in most cases after this change.

At the same time, SHALL in the RFC is not MUST (see https://datatracker.ietf.org/doc/html/rfc2119#section-3):

SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

I'm fine with making the change, but we should add an entry to the changelog, which explains that the change is backwards incompatible, and that Scrapy stops support some of these weird cases to improve the efficiency for the vast majority of cases. Users can always use json.loads(response.text) in their code, instead of the response.json() shortcut.

[jxlil]

Done! I've just added an entry to the Backward-incompatible changes section in the changelog. Please let me know if I need to correct something.

[Gallaecio]

SHALL != SHOULD.

MUST This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification.

[Gallaecio]

Looks good to me, provided the release notes fix.

[jxlil]

A test failed, but I think it's not related to the changes in this PR

[kmike]

Thanks @jxlil!