Skip to content
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python Other
Latest commit 4158839 @redapple redapple Merge pull request #1765 from scrapy/add-deprecation-for-pydispatch
[MRG+1] Add fallback and deprecation warning for pydispatch (fixes #1762)
Failed to load latest commit information.
artwork added artwork files properly now
debian Merge pull request #934 from Dineshs91/zsh-support
docs add deprecation for pydispatch (thanks for the help @redapple)
extras Merge pull request #934 from Dineshs91/zsh-support
scrapy Merge pull request #1765 from scrapy/add-deprecation-for-pydispatch
sep Spelling fixes
tests Merge pull request #1765 from scrapy/add-deprecation-for-pydispatch
.bumpversion.cfg Bump version: 1.2.0dev1 → 1.2.0dev2
.coveragerc Add coverage report trough codecov.io
.gitignore add coverage files to gitignore
.travis.yml Enable travis builds on tag patterns
AUTHORS added Nicolas Ramirez to AUTHORS
CODE_OF_CONDUCT.md Add Code of Conduct Version 1.3.0 from http://contributor-covenant.org/
CONTRIBUTING.md Put a blurb about support channels in CONTRIBUTING
INSTALL fix link to online installation instructions
LICENSE mv scrapy/trunk to root as part of svn2hg migration
MANIFEST.in ENH: include tests/ to source distribution in MANIFEST.in
Makefile.buildbot Generated version as pep440 and dpkg compatible
NEWS added NEWS file pointing to docs/news.rst
README.rst Add link to CoC mardown file on Github
conftest.py Simplify if statement
pytest.ini Don't collect tests by their class name
requirements-py3.txt raise minimal twisted version for py3
requirements.txt upgrade parsel and add shim for deprecated selectorlist methods
setup.cfg remove no longer existent examples from doc_files used in bdist_rpm. …
setup.py upgrade parsel and use its function to instantiate root for finding form
tox.ini Add Python 3.5 tox env and Python 3.4 tests in Travis CI

README.rst

Scrapy

PyPI Version PyPI Monthly downloads Build Status Wheel Status Python 3 Porting Status Coverage report

Overview

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Requirements

  • Python 2.7
  • Works on Linux, Windows, Mac OSX, BSD

Install

The quick way:

pip install scrapy

For more details see the install section in the documentation: http://doc.scrapy.org/en/latest/intro/install.html

Releases

You can download the latest stable and development releases from: http://scrapy.org/download/

Documentation

Documentation is available online at http://doc.scrapy.org/ and in the docs directory.

Community (blog, twitter, mail list, IRC)

See http://scrapy.org/community/

Contributing

Please note that this project is released with a Contributor Code of Conduct (see https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md).

By participating in this project you agree to abide by its terms. Please report unacceptable behavior to opensource@scrapinghub.com.

See http://doc.scrapy.org/en/master/contributing.html

Companies using Scrapy

See http://scrapy.org/companies/

Commercial Support

See http://scrapy.org/support/

Something went wrong with that request. Please try again.