An admin interface built with AngularJS to control Scrapy spiders, check their live stats, and get an overview of jobs. The EventSource of HTML5 Server-Sent Events is used to stream updates.
The project is just an interface. It depends on the pushserver project which uses flask-sse to stream updates.
Follow the instrunctions from pushserver and start your push server with
python manage.py runserver. This will start push server on http://localhost:8080/. This url is used when
initializing the EventSource.
Redis is also required to run. Install redis and run redis-server.
After you have successfully installed scrapyd, run scrapyd, which will
start the scrapy process listening on http://localhost:6800. This url is used in the
scrapyService to get
the data from Scrapy.
Finally run grunt server from scrapy-admin root directory to start the AngularJS project.
The whole project of course requires a working Scrapy projec which uses in some way flask-sse to send events. The configuration is done by the push server.