A fast high-level screen scraping and web crawling framework for Python.
Python Other
Pull request Compare This branch is 761 commits behind scrapy:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
artwork
debian
docs
extras
scrapy
sep
tests
.bumpversion.cfg
.coveragerc
.gitignore
.travis.yml
AUTHORS
CODE_OF_CONDUCT.md
CONTRIBUTING.md
INSTALL
LICENSE
MANIFEST.in
Makefile.buildbot
NEWS
README.rst
conftest.py
pytest.ini
requirements-py3.txt
requirements.txt
setup.cfg
setup.py
tox.ini

README.rst

Scrapy

PyPI Version Build Status Wheel Status Python 3 Porting Status Coverage report

Overview

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Requirements

  • Python 2.7 or Python 3.3+
  • Works on Linux, Windows, Mac OSX, BSD

Install

The quick way:

pip install scrapy

For more details see the install section in the documentation: http://doc.scrapy.org/en/latest/intro/install.html

Releases

You can download the latest stable and development releases from: http://scrapy.org/download/

Documentation

Documentation is available online at http://doc.scrapy.org/ and in the docs directory.

Community (blog, twitter, mail list, IRC)

See http://scrapy.org/community/

Contributing

Please note that this project is released with a Contributor Code of Conduct (see https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md).

By participating in this project you agree to abide by its terms. Please report unacceptable behavior to opensource@scrapinghub.com.

See http://doc.scrapy.org/en/master/contributing.html

Companies using Scrapy

See http://scrapy.org/companies/

Commercial Support

See http://scrapy.org/support/