Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Java Robot Exclusion Compliance Library

branch: master
README.md

JRobotX

Library to provide compliance with the Web Robot Exclusion protocol (robots.txt)

Usage

import com.trigonic.jrobotx.RobotExclusion;

// ...

RobotExclusion robotExclusion = new RobotExclusion();
if (robotExclusion.allows(url, userAgentString)) {
    // do something with url
}
Something went wrong with that request. Please try again.