Skip to content

Release: 0.0.1

Latest

Choose a tag to compare

@raj457036 raj457036 released this 19 Sep 18:13

Crawlers:

  • Async Crawler
  • Sync Crawler

Storages:

  • InMemoryStorage - for smaller websites
  • DiskStorage - for bigger websites

Configurable:

  • entrypoint
  • use host as entrypoint
  • max links to crawl
  • control ssl