a crawler case for xianyu base on pyspider
- crawl xianyu list page data(title,desc,userinfo,location,price...)
Use Python 3 and
- pip install pyspider
- run command pyspider, visit http://localhost:5000/
- create project
- Copy the script code in my_result_worker.py file to pyspider webdav mode
- Only tested on Mac OS X 10.13 and Debian 9
Downloading and viewing your data from WebUI is convenient, but may not suitable for computer. If want to store the cralwed date to local database, It's highly recommended to override ResultWorker. and you can use command-line to specify the parameters. A config file is a better choice.
For example command-line in my machine:
pyspider -c config.json --logging-config logging.conf