Java Robot Exclusion Compliance Library
Java Shell Groovy
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
gradle/wrapper
src
.gitignore
LICENSE.txt
README.md
build.gradle
gradlew
gradlew.bat
settings.gradle

README.md

JRobotX

Library to provide compliance with the Web Robot Exclusion protocol (robots.txt)

Usage

import com.trigonic.jrobotx.RobotExclusion;

// ...

RobotExclusion robotExclusion = new RobotExclusion();
if (robotExclusion.allows(url, userAgentString)) {
    // do something with url
}