Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python Other

Merge pull request #1476 from mlyundin/master

[MRG+1] Avoid creation of temporary list object in iflatten
latest commit 6ae8963256
@nramirezuy nramirezuy authored
Failed to load latest commit information.
artwork added artwork files properly now
debian Merge pull request #934 from Dineshs91/zsh-support
docs Merge pull request #1473 from ArturGaspar/master
extras Merge pull request #934 from Dineshs91/zsh-support
scrapy Merge pull request #1476 from mlyundin/master
sep mark SEP-019 as Final
tests Merge pull request #1473 from ArturGaspar/master
.bumpversion.cfg Bump version: 1.0.0rc1 → 1.1.0dev1
.coveragerc Add coverage report trough
.gitignore add coverage files to gitignore
.travis.yml Add coverage report trough
AUTHORS added Nicolas Ramirez to AUTHORS Put a blurb about support channels in CONTRIBUTING
INSTALL fix link to online installation instructions
LICENSE mv scrapy/trunk to root as part of svn2hg migration ENH: include tests/ to source distribution in
Makefile.buildbot Generated version as pep440 and dpkg compatible
NEWS added NEWS file pointing to docs/news.rst
README.rst Add coverage report trough remove scrapy.utils.testsite from PY3 ignores
pytest.ini Don't collect tests by their class name
requirements-py3.txt add service_identity to scrapy install_requires
requirements.txt upgrade parsel and add shim for deprecated selectorlist methods
setup.cfg remove no longer existent examples from doc_files used in bdist_rpm. … upgrade parsel and use its function to instantiate root for finding form
tox.ini Do not be verbose with coverage report by default



PyPI Version Build Status Wheel Status Python 3 Porting Status Coverage report


Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at:


  • Python 2.7
  • Works on Linux, Windows, Mac OSX, BSD


The quick way:

pip install scrapy

For more details see the install section in the documentation:


You can download the latest stable and development releases from:


Documentation is available online at and in the docs directory.

Community (blog, twitter, mail list, IRC)




Companies using Scrapy


Commercial Support


Something went wrong with that request. Please try again.