Update httpcache.py

[Kromitvs]

Sometimes (for instance, in sites finishing in '/', without 'html' at the end) the body is necessary to correctly identify a Response type as txt. Without passing this parameter, the body can never be used for that.

To do:

• scrapy/extensions/httpcache.py:242
• scrapy/extensions/httpcache.py:302
• scrapy/core/downloader/handlers/ftp.py:105
• scrapy/core/downloader/webclient.py:113

[Kromitvs]

Update httpcache.py

This is how it's already done when httpcache is not called:

scrapy/scrapy/core/downloader/handlers/http11.py

Line 436 in c292957

respcls = responsetypes.from_args(headers=headers, url=url, body=result["body"])

[codecov]

Codecov Report

Merging #4873 (0e39c2c) into master (de0e2cc) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 0e39c2c differs from pull request most recent head e1193ec. Consider uploading reports for the commit e1193ec to get more accurate results

@@           Coverage Diff           @@
##           master    #4873   +/-   ##
=======================================
  Coverage   88.71%   88.71%           
=======================================
  Files         162      162           
  Lines       10740    10743    +3     
  Branches     1834     1835    +1     
=======================================
+ Hits         9528     9531    +3     
  Misses        939      939           
  Partials      273      273           

Impacted Files	Coverage Δ
scrapy/core/downloader/handlers/ftp.py	98.38% <100.00%> (ø)
scrapy/core/downloader/webclient.py	94.65% <100.00%> (ø)
scrapy/extensions/httpcache.py	95.41% <100.00%> (ø)
scrapy/responsetypes.py	94.28% <100.00%> (+0.25%)	⬆️

[elacuesta]

It'd be great if this could be tested somehow, but it seems like a fairly harmless change so I'm approving anyway.

[Gallaecio]

@Kromitvs Do you think you can add a test for this?

[Kromitvs]

line 242 seems to have the same issue respcls = responsetypes.from_args(headers=headers, url=url)

There are a few more places were the call doesn't include the body, like scrapy/core/downloader/webclient.py:112. I don't know the code base that well to be sure.

I tested by scrapping the same site with and without cache. Are there other HttpCache tests I could get some inspiration from?

[Gallaecio]

HTTP cache tests are at https://github.com/scrapy/scrapy/blob/master/tests/test_downloadermiddleware_httpcache.py

If you find nothing there that helps figure out a way to test this, or if you are short on time, let me know; maybe I can have a stab at it.

[Kromitvs]

Hi, sorry to say, but it seems I can't find the time for now. Maybe I can help some other way or at some other time?

[kmike]

The PR looks good!