- All.txt - 18000+ sites compiled from alexa's topsites. This is the textfile
- that should work in scrapy as a test of production-readiness.
- test.txt - Custom page designed to be 3 pages deep, so that depth-limit can be
- verified as well as no-follow on non-domain pages.
- test_2.txt - Just google.com and mozilla.com, for a less controlled but larger
- potential manual test.