@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

  • Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 28,304 6,970 44 issues need help Updated Jul 21, 2018
  • Command line client for Scrapyd server

    Python 318 62 Updated Jul 20, 2018
  • The scrapy.org website

    HTML 28 72 Updated Jul 15, 2018
  • Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 332 53 Updated Jul 10, 2018
  • Python library of web-related functions

    Python 247 67 Updated Jul 2, 2018
  • A service daemon to run Scrapy spiders

    Python 1,239 354 Updated Jun 5, 2018
  • CSS Selectors for Python

    Python 179 39 Updated Jun 1, 2018
  • A CLI for benchmarking Scrapy.

    Python 16 6 MIT Updated May 22, 2018
  • Collection of persistent (disk-based) queues

    Python 152 36 Updated Mar 12, 2018
  • Fill HTML login forms automatically

    Python 198 55 Updated Dec 13, 2017
  • This is a sample Scrapy project for educational purposes

    Python 484 353 MIT Updated Oct 31, 2017
  • Scrapy project to scrape public web directories (educational) [DEPRECATED]

    Python 1,512 1,142 Updated Oct 27, 2017
  • A pure-python HTML screen-scraping library

    HTML 1,464 222 Updated Oct 20, 2017
  • Codespeed for scrapy-bench

    Python 1 1 Updated Aug 28, 2017
  • A fork of http://pydispatcher.sourceforge.net/ with PyPy support

    Python 6 1 Updated Jul 3, 2017
  • Python 6 122 Updated Mar 31, 2017
  • Library to populate Scrapy items using XPath and CSS with a convenient API

    Python 4 5 BSD-3-Clause Updated Jul 26, 2016
  • 224 57 Updated Apr 27, 2015
  • GSoC2014 - Scrapy Integration tests project

    Shell 3 1 Updated Mar 18, 2014