Crappyspider is a generic cralwer. The goal is to stay kiss and simple, test easily your site. A report is generate with all visited url and can be used by other tool to make functional test.
The project is available on pypi:
pip install crappyspider
Or from source:
python setup.py install
Then you can run a spider:
scrapy crawl crappyspider -a config=my_rule.json
See http://crappyspider.readthedocs.org/en/latest/ for further information.