分布式爬虫,爬取京东在售显卡信息
使用scrapy_redis实现scrapy各个组件与redis交互
实现指纹去重集合;持久化任务队列
实现断点续爬和分布式爬虫
-
Notifications
You must be signed in to change notification settings - Fork 0
lonerge/redis_and_scrapy_scrapy_redis
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
分布式爬虫,爬取京东在售显卡信息
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published