#
robotstxt
Here are 3 public repositories matching this topic...
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
-
Updated
Nov 9, 2021 - Rust
A native Rust port of Google's robots.txt parser and matcher C++ library.
-
Updated
Feb 13, 2021 - Rust
Improve this page
Add a description, image, and links to the robotstxt topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robotstxt topic, visit your repo's landing page and select "manage topics."