Lightweight robots.txt parser and generator written in Rust.
-
Updated
Sep 19, 2020 - Rust
Lightweight robots.txt parser and generator written in Rust.
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."