(C)oncurrent (RUST) Crawl(er) How to use Edit config.toml with desired starting url cargo run Todo Fix too many files open error Add ability to download pdfs Add ability to filter when crawling Archive crawled urls Add queue system Add option to archive in different datastores Dockerize