BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.
Scrapy provides a built-in mechanism for extracting data (called selectors <topics-selectors>
) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they're just parsing libraries which can be imported and used from any Python code.
In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django.
Scrapy runs in Python 2.5, 2.6 and 2.7. But it's recommended you use Python 2.6 or above, since the Python 2.5 standard library has a few bugs in their URL handling libraries. Some of these Python 2.5 bugs not only affect Scrapy but any user code, such as spiders.
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment, Scrapy works with Python 2.5, 2.6 and 2.7.
faq-python-versions
.
Probably, but we don't like that word. We think Django is a great open source project and an example to follow, so we've used it as an inspiration for Scrapy.
We believe that, if something is already done well, there's no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We'd be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See ~scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware
.
You need to install pywin32 because of this Twisted bug.
See topics-request-response-ref-request-userlogin
.
By default, Scrapy uses a LIFO queue for storing pending requests, which basically means that it crawls in DFO order. This order is more convenient in most cases. If you do want to crawl in true BFO order, you can do it by setting the following settings:
DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
See topics-leaks
.
Also, Python has a builtin memory leak issue which is described in topics-leaks-without-leaks
.
See previous question.
Yes, see ~scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware
.
Try changing the default Accept-Language request header by overriding the DEFAULT_REQUEST_HEADERS
setting.
See intro-examples
.
Yes. You can use the runspider
command. For example, if you have a spider written in a my_spider.py
file you can run it with:
scrapy runspider my_spider.py
See runspider
command for more info.
Those messages (logged with DEBUG
level) don't necessarily mean there is a problem, so you may not need to fix them.
Those message are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.
For more info see: ~scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware
.
See topics-scrapyd
.
It'll depend on how large your output is. See this warning
<json-with-large-data>
in ~scrapy.contrib.exporter.JsonItemExporter
documentation.
Some signals support returning deferreds from their handlers, others don't. See the topics-signals-ref
to know which ones.
999 is a custom reponse status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2
(or higher) in your spider:
class MySpider(CrawlSpider):
name = 'myspider'
DOWNLOAD_DELAY = 2
# [ ... rest of the spider code ... ]
Or by setting a global download delay in your project with the DOWNLOAD_DELAY
setting.
Yes, but you can also use the Scrapy shell which allows you too quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than plain old pdb.set_trace()
.
For more info see topics-shell-inspect-response
.
To dump into a JSON file:
scrapy crawl myspider -o items.json -t json
To dump into a CSV file:
scrapy crawl myspider -o items.csv -t csv
To dump into a XML file:
scrapy crawl myspider -o items.xml -t xml
For more information see topics-feed-exports
The __VIEWSTATE
parameter is used in sites built with ASP.NET/VB.NET. For more info on how it works see this page. Also, here's an example spider which scrapes one of these sites.
Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory.
In order to avoid parsing all the entire feed at once in memory, you can use the functions xmliter
and csviter
from scrapy.utils.iterators
module. In fact, this is what the feed spiders (see topics-spiders
) use under the cover.
Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them back on subsequent requests, like any regular web browser does.
For more info see topics-request-response
and cookies-mw
.
Enable the COOKIES_DEBUG
setting.
Raise the ~scrapy.exceptions.CloseSpider
exception from a callback. For more info see: ~scrapy.exceptions.CloseSpider
.
Some websites implement certain measures to prevent bots from crawling them, with varying degrees of sophistication. Getting around those measures can be difficult and tricky, and may sometimes require special infrastructure.
Here are some tips to keep in mind when dealing with these kind of sites:
- rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them)
- disable cookies (see
COOKIES_ENABLED
) as some sites may use cookies to spot bot behaviour - use download delays (2 or higher). See
DOWNLOAD_DELAY
setting. - is possible, use Google cache to fetch pages, instead of hitting the sites directly
- use a pool of rotating IPs. For example, the free Tor project.
If you are still unable to prevent your bot getting banned, consider contacting commercial support.