Scrapyd is a service for running Scrapy spiders.
It allows you to deploy your Scrapy projects and control their spiders using an HTTP JSON API.
The documentation (including installation and usage) can be found at: http://scrapyd.readthedocs.org/
To save scrapyd cache to file add to scrapyd.conf: cache_dbs = dbs/cache.db