[MRG+1] Is_gzipped for application/x-gzip;charset=utf-8

[Tethik]

Fix for #2049

[codecov-io]

Current coverage is 83.34%

Merging #2050 into master will increase coverage by <.01%

Powered by Codecov. Last updated by b7925e4...23f99e9

[robsonpeixoto]

Just a suggestion:

from six.moves import map # just if you would like a lazy solution

mimes = (b'application/x-gzip', b'application/gzip')
return any(map(lambda mime: mime in ctype, mimes))

[Tethik]

I did it as an any-statement first, but figured it would be overkill since there are only two comparisons. Another version:

any(mime in ctype for mime in (b'application/x-gzip', b'application/gzip'))

[robsonpeixoto]

I do not known about the scrapy code style, but IMO it should be 4 different tests.

[redapple]

According to RFC 7231, section-3.1.1.1,

The type, subtype, and parameter name tokens are case-insensitive.

so it's worth fixing that too.

Suggestion:

$ git diff
diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py
index f174950..e520b42 100644
--- a/scrapy/utils/gz.py
+++ b/scrapy/utils/gz.py
@@ -1,3 +1,4 @@
+import re
 import struct

 try:
@@ -51,7 +52,9 @@ def gunzip(data):
     return output


+_is_gzipped_re = re.compile(br'^application/(x-)?gzip\b', re.I)
+
 def is_gzipped(response):
     """Return True if the response is gzipped, or False otherwise"""
     ctype = response.headers.get('Content-Type', b'')
-    return b'application/x-gzip' in ctype or b'application/gzip' in ctype
+    return _is_gzipped_re.search(ctype)
diff --git a/tests/test_utils_gz.py b/tests/test_utils_gz.py
index 3648d5c..02bf1d7 100644
--- a/tests/test_utils_gz.py
+++ b/tests/test_utils_gz.py
@@ -37,6 +37,16 @@ class GunzipTest(unittest.TestCase):
         r1 = Response("http://www.example.com", headers=hdrs)
         self.assertTrue(is_gzipped(r1))

+    def test_is_gzipped_case_insensitive(self):
+        hdrs = Headers({"Content-Type": "Application/X-Gzip"})
+        r1 = Response("http://www.example.com", headers=hdrs)
+        self.assertTrue(is_gzipped(r1))
+
+    def test_is_gzipped_not_quite(self):
+        hdrs = Headers({"Content-Type": "application/gzippppp"})
+        r1 = Response("http://www.example.com", headers=hdrs)
+        self.assertFalse(is_gzipped(r1))
+
     def test_is_gzipped_empty(self):
         r1 = Response("http://www.example.com")
         self.assertTrue(not is_gzipped(r1))
@@ -50,3 +60,7 @@ class GunzipTest(unittest.TestCase):
         hdrs = Headers({"Content-Type": "application/x-gzip;charset=utf-8"})
         r1 = Response("http://www.example.com", headers=hdrs)
         self.assertTrue(is_gzipped(r1))
+
+        hdrs = Headers({"Content-Type": "application/X-GZIP ; charset=utf-8"})
+        r1 = Response("http://www.example.com", headers=hdrs)
+        self.assertTrue(is_gzipped(r1))

[Tethik]

@redapple Alright, I'll look into it. Good catch on the gzipppp

[Tethik]

Done

[redapple]

I usually prefer return _is_gzipped_re.search(ctype) is not None

[Tethik]

Zzzz... fine ;)

[redapple]

Sorry for nitpicking, but I first read (quickly) "not is_gzipped".

[Tethik]

Nahh it's fine. I agree is not looks better. I'm just lazy :)

[redapple]

Looks good to me.
Thanks @Tethik!

[robsonpeixoto]

IMO should break this test in two different:

def test_is_gzipped_right(self):
    hdrs = Headers({"Content-Type": "application/gzip"})
    r1 = Response("http://www.example.com", headers=hdrs)
    self.assertTrue(is_gzipped(r1))

def test_is_x_gzipped_right(self):
    hdrs = Headers({"Content-Type": "application/x-gzip"})
    r1 = Response("http://www.example.com", headers=hdrs)
    self.assertTrue(is_gzipped(r1))