jeremyf / robot_rules

A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.

The robot_rules network graph

All branches in the network using jeremyf/robot_rules as the reference point. Read our blog post about how it works.

Show Help

This is the first time this graph has been requested. It may take a few moments to calculate and cache the data.