I had a site return the following header, which caused SitemapSpider to not parse a sitemap xml that was gzipped.
Content-Type: application/x-gzip;charset=utf-8
Looking into the code it seems that the function utils.is_gzipped does not take into account cases where the Content-Type header would include charset.
The text was updated successfully, but these errors were encountered:
I had a site return the following header, which caused SitemapSpider to not parse a sitemap xml that was gzipped.
Looking into the code it seems that the function utils.is_gzipped does not take into account cases where the Content-Type header would include charset.
The text was updated successfully, but these errors were encountered: