while trying to access sitemap from robots.txt , Scrapy fails with IOError, 'Not a gzipped file' error
i did download file manually and was able to extract the content so it is not like file is corrupted
as an example sitemap url : you can follow amazon robots.txt
The text was updated successfully, but these errors were encountered:
Thanks @DharmeshPandav for reminding us of this issue.
from the web server, with
When reading IANA's page on
I'd go for considering