Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Java Robot Exclusion Compliance Library
Java Shell Groovy
branch: master

Merge pull request #4 from AlanKrueger/master

update to gradle 1.2 and add nexus plugin
latest commit 5354050843
@AlanKrueger AlanKrueger authored

README.md

JRobotX

Library to provide compliance with the Web Robot Exclusion protocol (robots.txt)

Usage

import com.trigonic.jrobotx.RobotExclusion;

// ...

RobotExclusion robotExclusion = new RobotExclusion();
if (robotExclusion.allows(url, userAgentString)) {
    // do something with url
}
Something went wrong with that request. Please try again.