install

$ pip install -r requiremets.txt

single url crawling

$ python crowl.py --conf config.ini

$ python crowl_multiple.py --conf config.ini --index 1

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
crawls		crawls
.gitignore		.gitignore
LICENSE.txt		LICENSE.txt
README.md		README.md
crowl.py		crowl.py
crowl_multiple.py		crowl_multiple.py
foo.txt		foo.txt
index.txt		index.txt
items.py		items.py
pipelines.py		pipelines.py
requirements.txt		requirements.txt
spiders.py		spiders.py
test-urls.txt		test-urls.txt
test-urls2.txt		test-urls2.txt
utils.py		utils.py