Skip to content

Latest commit

 

History

History
75 lines (65 loc) · 3.52 KB

EN_README.md

File metadata and controls

75 lines (65 loc) · 3.52 KB
Raw

SpiderMan

English | 中文

table of Contents

statement

SpiderManpages are a lot of use 百度图片Untagged copyrighted pictures, please contact me if you are suspected of infringement.

BackgroundIntroduction

SpiderMan Based on Scrapy, scrapyd, scrapy-API, tornado spider distributed management framework.

SpiderMan Features include spider scheduling, Web end code editor and Scrapy project construction and so on.

ProjectIntroduction

SpiderMan is designed to provide a convenient and distributed crawler management framework

SpiderMan is developed on the basis of scrapyd API, and does not make any intrusion to existing crawler code
SpiderMan is embedded in the web editor so that you can write code, publish code, deploy new crawler projects to a specified server in web pages
SpiderMan At present, only tornado.ioloop.PeriodicCallback is used to complete the timing task, and a single spider is scheduled for regular scheduling

Instructions

GetCode

UseAnEexample

InterfaceIntroduction

  • HOME: Because I really don't know what to put a few things on the front page, but if there is no home page, I always feel strange!
    Shurnim icon
  • SERVER: It is mainly used for the management of scrapyd servers, including: adding new servers, scheduling server specified spiders, deleting the items of the designated servers of the server.
  • PROJECT: The new local project is deployed to the specified server. This page includes: adding new projects, deleting items, editing project code, and deploying projects
  • There are also some subpages...

InterfaceStyle

  • HOME
    Shurnim icon
  • SERVER
    Shurnim icon
  • PROJECT
    Shurnim icon

deploy

clone project to local
git clone https://github.com/PerrorOne/SpiderMan.git
cd SpiderMan
python setup.py install
# init project
SpiderMan init
# Appoint host,port 
SpiderMan runserver [host:port]
# Create an administrator account
SpiderMan admin
#=================Use docker======================
git clone https://github.com/PerrorOne/SpiderMan.git
cd SpiderMan
docker build -t spiderman:v1 .
docker run -it -p 8080:8080 spiderman:v1

Other

Because my programming level is limited, and I am developing alone,
the page is not internationalized, the timing task is too simple, and so on. A series of problems,
if friends have better ideas, better implementations, or find bug. welcome fork modification!
Finally, I wish you a happy life!
Shurnim icon