[MRG] Always decompress Content-Encoding: gzip at HttpCompression stage

[redapple]

Let SitemapSpider handle decoding of .xml.gz files if necessary.

Fixes #2389

The change here is to always decompress responses with Content-Encoding: gzip, whatever Content-Type says, and contrary to #193 (comment), #660, #2065

Therefore, SitemapSpider still has to decode "real" .gz files/content if the HTTP response was not "Content-Encoded", relying on gzip magic number instead of response headers.

I believe it follows RFC 7231 better:

The "Content-Encoding" header field indicates what content codings
have been applied to the representation, beyond those inherent in the
media type, and thus what decoding mechanisms have to be applied in
order to obtain data in the media type referenced by the Content-Type
header field. Content-Encoding is primarily used to allow a
representation's data to be compressed without losing the identity of
its underlying media type.

#951 (comment) also hinted at something like this.

I think it can also fix #2162 though I need to test that.

[codecov-io]

Codecov Report

Merging #2391 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2391      +/-   ##
==========================================
+ Coverage   84.15%   84.16%   +0.01%     
==========================================
  Files         162      162              
  Lines        9079     9079              
  Branches     1346     1345       -1     
==========================================
+ Hits         7640     7641       +1     
  Misses       1177     1177              
+ Partials      262      261       -1

Impacted Files	Coverage Δ
scrapy/spiders/sitemap.py	56.66% <100%> (-1.4%)	❌
scrapy/downloadermiddlewares/httpcompression.py	88.88% <100%> (+2.22%)	✅
scrapy/utils/gz.py	100% <100%> (ø)	✅

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6320b74...b6378c7. Read the comment docs.

[sibiryakov]

Should we make this behavior optional @redapple ?

[redapple]

@sibiryakov , I would prefer that we make this change not configurable, handing responses with the correct type sooner than later.
#2393 is a good companion to this

[kmike]

https://mimesniff.spec.whatwg.org/#matching-an-archive-type-pattern suggests a 3-byte signature

[redapple]

Makes sense @kmike , thanks for the pointer. PR updated.

[kmike]

Other than gzip signature this PR looks good. Signature also looks fine - 2 bytes is what gzip format rfc's say, and what wikipedia shows, but 3 bytes with hardcoded compression method seems to be more robust and recommended in mime sniffing spec.

@sibiryakov it seems the only case HTTP decompression should be made optional is when response bodies are not processed at all (no link extraction, no peeking into response body), and it is fine to store them in a raw compressed form, as they are received from transport - it means e.g. removing an ability to search inside these bodies. For me it sounds like a rare use case (http cache without any processing? what to use these cache results for?) - if you're downloading a page then likely you want to use it somehow, and to do that you need to decompress it.

This issue is not about decompressing all gzipped files automatically (scrapy is not doing this), it is about undoing http compression.

[kmike]

pytest magic asserts are disabled in scrapy testing suite (they break contracts tests if I recall it correctly), so it is better to keep assertIs / assertEqual

[redapple]

Done.

[kmike]

Thanks @redapple!