v0.1.0 - First Draft of Crawler Framework for Python
First draft
-
Components:
Different components response of different things.
1-1. HTTP request sender
1-2. HTTP response parser
1-3. Data processing handler
1-4. Persistence layer -
Crawler roles:
Different crawler role works in different way.
2-1. SimpleCrawler
2-2. AsyncSimpleCrawler
2-3. ExecutorCrawler
2-4. PoolCrawler