Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported
Switch branches/tags
Nothing to show
Clone or download

README.md

ScrapydWeb: Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported

Features

  • Multinode Scrapyd Servers

    • Group, filter and select any numbers of nodes
    • Execute command on multinodes with one click
  • Scrapy Log Analysis

    • Stats collection
    • Progress visualization
    • Logs categorization
  • All Scrapyd API Supported

    • Deploy project, Run Spider, Stop job
    • List projects/versions/spiders/running_jobs
    • Delete version/project
  • Others

    • Basic auth for web UI
    • Accessing Scrapyd servers protected by basic auth

Maintainer

Installation

To install ScrapydWeb, simply use pip:

pip install scrapydweb

Start Up

  1. Run scrapydweb -h to get help, and a config file named scrapydweb_settings.py would be copied to the working directory, then you can custom config with it.
  2. Run scrapydweb
  3. Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)

Screenshot

  • Overview overview

  • Dashboard dashboard

  • Log Analysis

    • Stats collection log_stats

    • Progress visualization log_chart

    • Logs categorization log_extracted

  • Deploy Project deploy

  • Run Spider run

  • Manage Projects manage