Skip to content

oglinuk/cruster

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

(C)oncurrent (RUST) Crawl(er)

How to use

  • Edit config.toml with desired starting url
  • cargo run

Todo

  • Fix too many files open error
  • Add ability to download pdfs
  • Add ability to filter when crawling
  • Archive crawled urls
  • Add queue system
  • Add option to archive in different datastores
  • Dockerize

About

Concurrent web crawler ported from Go to Rust

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages