Skip to content
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python Other
  1. Python 99.8%
  2. Other 0.2%
Branch: master
Clone or download
Latest commit 1364e92 Sep 20, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE Add Bug report and Feature request templates (#3471) Aug 26, 2019
artwork Use https for external links wherever possible in docs Oct 26, 2017
debian Remove deprecated xlib module Sep 13, 2019
docs Merge pull request #4016 from elacuesta/rule_default_link_extractor Sep 20, 2019
extras Use https for external links wherever possible in docs Oct 26, 2017
scrapy Merge pull request #4015 from elacuesta/remove_deprecated_xlib Sep 20, 2019
sep Merge branch 'master' into backward Mar 14, 2019
tests Merge pull request #4015 from elacuesta/remove_deprecated_xlib Sep 20, 2019
.bumpversion.cfg Bump version: 1.6.0 → 1.7.0 Jul 18, 2019
.coveragerc Remove deprecated xlib module Sep 13, 2019
.gitignore remove duplicated entry in gitignore Mar 22, 2019
.travis.yml [MRG+1] Drop py34 support - Update CI envs (#3892) Aug 7, 2019
AUTHORS added Nicolas Ramirez to AUTHORS Mar 14, 2013
CODE_OF_CONDUCT.md minor grammatical fixes in CODE_OF_CONDUCT.md Jun 2, 2018
CONTRIBUTING.md Be consistent with domain used for links to documentation website Jan 31, 2019
INSTALL Be consistent with domain used for links to documentation website Jan 31, 2019
LICENSE added oxford commas to LICENSE Jun 2, 2018
MANIFEST.in Include additional files in sdists Nov 16, 2018
Makefile.buildbot Generated version as pep440 and dpkg compatible Jun 16, 2015
NEWS added NEWS file pointing to docs/news.rst Apr 29, 2012
README.rst [MRG+1] Drop py34 support - Update CI envs (#3892) Aug 7, 2019
appveyor.yml Set the cloned directory as PYTHONPATH in appveyor.yml Jun 11, 2019
codecov.yml codecov config: disable project check, tweak PR comments May 18, 2017
conftest.py Remove deprecated xlib module Sep 13, 2019
pytest.ini Don't collect tests by their class name May 4, 2015
requirements-py2.txt [MRG+1] Drop py34 support - Update CI envs (#3892) Aug 7, 2019
requirements-py3.txt [MRG+1] Drop py34 support - Update CI envs (#3892) Aug 7, 2019
setup.cfg Build universal wheels Mar 1, 2016
setup.py [MRG+1] Drop py34 support - Update CI envs (#3892) Aug 7, 2019
tox.ini Adds integration with Protego robots.txt parser (#3935) Aug 27, 2019

README.rst

Scrapy

PyPI Version Supported Python Versions Build Status Wheel Status Coverage report Conda Version

Overview

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: https://scrapy.org

Requirements

  • Python 2.7 or Python 3.5+
  • Works on Linux, Windows, Mac OSX, BSD

Install

The quick way:

pip install scrapy

For more details see the install section in the documentation: https://docs.scrapy.org/en/latest/intro/install.html

Documentation

Documentation is available online at https://docs.scrapy.org/ and in the docs directory.

Releases

You can find release notes at https://docs.scrapy.org/en/latest/news.html

Community (blog, twitter, mail list, IRC)

See https://scrapy.org/community/

Contributing

See https://docs.scrapy.org/en/master/contributing.html

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct (see https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md).

By participating in this project you agree to abide by its terms. Please report unacceptable behavior to opensource@scrapinghub.com.

Companies using Scrapy

See https://scrapy.org/companies/

Commercial Support

See https://scrapy.org/support/

You can’t perform that action at this time.