Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python Other

Merge pull request #1476 from mlyundin/master

[MRG+1] Avoid creation of temporary list object in iflatten
latest commit 6ae8963256
@nramirezuy nramirezuy authored
Failed to load latest commit information.
artwork added artwork files properly now
debian Merge pull request #934 from Dineshs91/zsh-support
docs Merge pull request #1473 from ArturGaspar/master
extras Merge pull request #934 from Dineshs91/zsh-support
scrapy Merge pull request #1476 from mlyundin/master
sep mark SEP-019 as Final
tests Merge pull request #1473 from ArturGaspar/master
.bumpversion.cfg Bump version: 1.0.0rc1 → 1.1.0dev1
.coveragerc Add coverage report trough codecov.io
.gitignore add coverage files to gitignore
.travis.yml Add coverage report trough codecov.io
AUTHORS added Nicolas Ramirez to AUTHORS
CONTRIBUTING.md Put a blurb about support channels in CONTRIBUTING
INSTALL fix link to online installation instructions
LICENSE mv scrapy/trunk to root as part of svn2hg migration
MANIFEST.in ENH: include tests/ to source distribution in MANIFEST.in
Makefile.buildbot Generated version as pep440 and dpkg compatible
NEWS added NEWS file pointing to docs/news.rst
README.rst Add coverage report trough codecov.io
conftest.py remove scrapy.utils.testsite from PY3 ignores
pytest.ini Don't collect tests by their class name
requirements-py3.txt add service_identity to scrapy install_requires
requirements.txt upgrade parsel and add shim for deprecated selectorlist methods
setup.cfg remove no longer existent examples from doc_files used in bdist_rpm. …
setup.py upgrade parsel and use its function to instantiate root for finding form
tox.ini Do not be verbose with coverage report by default

README.rst

Scrapy

PyPI Version Build Status Wheel Status Python 3 Porting Status Coverage report

Overview

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Requirements

  • Python 2.7
  • Works on Linux, Windows, Mac OSX, BSD

Install

The quick way:

pip install scrapy

For more details see the install section in the documentation: http://doc.scrapy.org/en/latest/intro/install.html

Releases

You can download the latest stable and development releases from: http://scrapy.org/download/

Documentation

Documentation is available online at http://doc.scrapy.org/ and in the docs directory.

Community (blog, twitter, mail list, IRC)

See http://scrapy.org/community/

Contributing

See http://doc.scrapy.org/en/master/contributing.html

Companies using Scrapy

See http://scrapy.org/companies/

Commercial Support

See http://scrapy.org/support/

Something went wrong with that request. Please try again.