GitHub - jiandanshihun/pycraw: A little python crawler. Thanks for the powerful scrapy.

#Python Crawler#

Usage: python mini_spider.py <start url file> -c conf

Start url file:

conf file:

Notice:

If you want something useful to store, just try to explore the processor:item

Wish You Good Luch

downtownguy.hui@gmail.com

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
common		common
extractor		extractor
log		log
processor		processor
spider		spider
test		test
tools		tools
Readme.md		Readme.md
controller.py		controller.py
extractor_manager.py		extractor_manager.py
log.py		log.py
mini_spider.conf		mini_spider.conf
mini_spider.py		mini_spider.py
processor_manager.py		processor_manager.py
safe_dict.py		safe_dict.py
spider_manager.py		spider_manager.py