Alternative robots parser module for Python
-
Updated
Aug 1, 2024 - Python
Alternative robots parser module for Python
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
Robots Scanner
python binding for Google robots.txt parser C++ library
Add a description, image, and links to the robotstxt topic page so that developers can more easily learn about it.
To associate your repository with the robotstxt topic, visit your repo's landing page and select "manage topics."