Robots.txt parser and generator - Work in progress
-
Updated
Apr 26, 2018 - Go
Robots.txt parser and generator - Work in progress
Go language library for parsing Sitemaps
Package robots implements robots.txt file parsing and matching based on Google's specification.
Enumerate old versions of robots.txt paths using Wayback Machine for content discovery
grobotstxt is a native Go port of Google's robots.txt parser and matcher library.
Go robots.txt parser
The robots.txt exclusion protocol implementation for Go language
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."