Skip to content

v0.1.0 - First Draft of Crawler Framework for Python

Compare
Choose a tag to compare
@Chisanan232 Chisanan232 released this 20 Jan 14:52
· 168 commits to master since this release

First draft


  1. Components:
    Different components response of different things.
    1-1. HTTP request sender
    1-2. HTTP response parser
    1-3. Data processing handler
    1-4. Persistence layer

  2. Crawler roles:
    Different crawler role works in different way.
    2-1. SimpleCrawler
    2-2. AsyncSimpleCrawler
    2-3. ExecutorCrawler
    2-4. PoolCrawler