v0.1.0

[0.1.0] - 2021-10-07

Added

• Parsing robots.txt files
  • user-agent groups
  • allow and disallow rule patterns
• Check if a certain uri/path is allowed for a certain bot/user-agent according to the definitions in robots.txt
• Implemented as described in https://datatracker.ietf.org/doc/html/draft-koster-rep