Skip to content

AlanKrueger/jrobotx

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JRobotX

Library to provide compliance with the Web Robot Exclusion protocol (robots.txt)

Usage

import com.trigonic.jrobotx.RobotExclusion;

// ...

RobotExclusion robotExclusion = new RobotExclusion();
if (robotExclusion.allows(url, userAgentString)) {
    // do something with url
}

About

Java Robot Exclusion Compliance Library

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 85.8%
  • Shell 10.1%
  • Groovy 4.1%