Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉

admin ui for scrapy/open source scrapinghub

豆瓣电影/豆瓣读书 Scarpy 爬虫

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

狠心开源企业级舆情新闻爬虫项目：支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除；爬虫一键部署；爬虫监控可视化; 配置集群爬虫分配策略；👉 现成的docker一键部署文档已为大家踩坑

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

Wrapper for running Scrapyd in Heroku or locally as a service

How to set up Scrapyd cluster on Heroku

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

admin ui for scrapy/open source scrapinghub

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

How to set up Scrapyd cluster on Heroku

A django admin site for scrapy

Simple Web UI for Scrapy spider management via Scrapyd

爬虫管理平台

Library designed to replace the SQLite backend by a MongoDB backend on Scrapy queue management

Implement scrapy with asyncio