Rust Web Crawler A very simple web crawler in rust. Usage cargo run <url to crawl> The crawler will then check the robots.txt and crawl the site if allowed.