模仿scrapy的轻量级爬虫框架
- 支持中间件，可以从
@from_crawler中读取配置。 - 实现断点续爬/爬虫中断之后恢复爬取队列和记录已爬取的队列
- 合并ORM到爬虫
- 实现批量入库
- 支持断线重连
-
Notifications
You must be signed in to change notification settings - Fork 2
xiantang/mini_scrapy
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|