Scrapy Tips

• Official Document
• 使用scrapy抓取网站

Ban

• 如何让你的scrapy爬虫不再被ban

Export result

Save File

• How to download a file with Scrapy?

UTF8

• https://segmentfault.com/q/1010000000367894
• http://stackoverflow.com/questions/30948736/unicode-on-scrapy-json-output
• http://git.oschina.net/ldshuang/imax-spider/commit/1d05d7bafdf7758f7b422cc1133abf493bf55086

Problems

Stop Crawling

• http://stackoverflow.com/questions/29844131/how-to-stop-scrapy-crawler
• http://stackoverflow.com/questions/4448724/force-my-scrapy-spider-to-stop-crawling

'exceptions.KeyError' in CrawlSpider

• http://stackoverflow.com/questions/23184688/scrapy-exceptions-keyerror-in-crawlspider

Connection was refused by other side: 111

• scrapy爬虫出现connection refused的一个可能原因
• http://stackoverflow.com/questions/5014980/scrapy-connection-refused