robots.txt Tester App

An App for testing URLs and User-agents blocked by robots.txt Files.

Given a certain robots.txt file, check whether or not any of the provided URLs can be fetched by all the user-agents in the robots.txt file (checks all the combinations).

Start testing!