Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python Other

Merge pull request #1527 from scrapy/no-pillow-300

TST don't use broken Pillow version in tests
latest commit 9676cf8e53
@kmike kmike authored
Failed to load latest commit information.
artwork added artwork files properly now
debian Merge pull request #934 from Dineshs91/zsh-support
docs fix another invalid xpath error
extras Merge pull request #934 from Dineshs91/zsh-support
scrapy Merge pull request #1498 from Preetwinder/scrapy-add_scheme
sep mark SEP-019 as Final
tests Restructure tests for add_http_if_no_scheme function
.bumpversion.cfg Bump version: 1.0.0rc1 → 1.1.0dev1
.coveragerc Add coverage report trough
.gitignore add coverage files to gitignore
.travis.yml don't run tests twice on Travis if a PR is made from a scrapy/scrapy …
AUTHORS added Nicolas Ramirez to AUTHORS Put a blurb about support channels in CONTRIBUTING
INSTALL fix link to online installation instructions
LICENSE mv scrapy/trunk to root as part of svn2hg migration ENH: include tests/ to source distribution in
Makefile.buildbot Generated version as pep440 and dpkg compatible
NEWS added NEWS file pointing to docs/news.rst
README.rst Add PyPI download stats badge remove scrapy.utils.testsite from PY3 ignores
pytest.ini Don't collect tests by their class name
requirements-py3.txt add service_identity to scrapy install_requires
requirements.txt upgrade parsel and add shim for deprecated selectorlist methods
setup.cfg remove no longer existent examples from doc_files used in bdist_rpm. … upgrade parsel and use its function to instantiate root for finding form
tox.ini TST don't use broken Pillow version in tests



PyPI Version PyPI Monthly downloads Build Status Wheel Status Python 3 Porting Status Coverage report


Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at:


  • Python 2.7
  • Works on Linux, Windows, Mac OSX, BSD


The quick way:

pip install scrapy

For more details see the install section in the documentation:


You can download the latest stable and development releases from:


Documentation is available online at and in the docs directory.

Community (blog, twitter, mail list, IRC)




Companies using Scrapy


Commercial Support


Something went wrong with that request. Please try again.