rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
-
Updated
Nov 9, 2021 - Rust
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
Lightweight robots.txt parser and generator written in Rust.
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."