Google Robots.txt Parser and Matcher Library in Java

This project aims to implement the robots.txt parser and matcher in Java. It is based on the C++ implementation.

About the library

The Robots Exclusion Protocol (REP) is a standard that enables website owners to control which URLs may be accessed by automated clients (i.e. crawlers) through a simple text file with a specific syntax. It's one of the basic building blocks of the internet as we know it and what allows search engines to operate.

Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to confusion. This project aims to fix that by releasing the parser that Google uses.

The library is a Java port of C++ parser and matcher which is a slightly modified production code used by Googlebot, Google's crawler. The library is released open-source to help developers build tools that better reflect Google's robots.txt parsing and matching.

For webmasters, we included a runnable class RobotsParserApp which is a small application that allows testing a single URL and several user-agents against a robots.txt.

Development

Prerequisites

You need Maven to build this project. Download and install it from the official website.

You can also install it like this if your Linux supports it:

$ sudo apt-get install maven

Build it

Using Maven

Standard maven commands work here.

$ mvn install

Or if you want a build from scratch:

$ mvn clean install

Using Maven Assembly Plugin

Alternatively, you can compile the entire project into a single JAR using the following command:

$ mvn clean compile assembly:single

You can find the result in target directory.

Run it

Using Maven

Following commands will run an application that parses given robots.txt file and print a matching verdict: ALLOWED or DISALLOWED (exit codes are 0 and 1 respectively).

You should provide a target URL using -u (--url) flag. At least one agent must be specified using -a (--agent) flag (verdict DISALLOWED is printed iff none of the user-agents are allowed to crawl given URL).

When flag -f (--file) is omitted, robots.txt contents are expected to be received via standard input:

$ mvn exec:java -Dexec.mainClass=com.google.search.robotstxt.RobotsParserApp -Dexec.args="--agent FooBot --url http://foo.com/bar"

If you want the application to read an existing robots.txt file, use flag -f (--file):

$ mvn exec:java -Dexec.mainClass=com.google.search.robotstxt.RobotsParserApp -Dexec.args="--agent FooBot --url http://foo.com/bar --file path/to/robots.txt"

From JAR

If you have built the project into JAR, you can run it from there (reading robots.txt from standard input):

$ java -jar target/robotstxt-java-1.0-SNAPSHOT-jar-with-dependencies.jar --agent FooBot --url http://foo.com/bar

Or (reading from file):

$ java -jar target/robotstxt-java-1.0-SNAPSHOT-jar-with-dependencies.jar --agent FooBot --url http://foo.com/bar --file path/to/robots.txt

Notes

Parsing of robots.txt files themselves is done exactly as in the production version of Googlebot, including how percent codes and unicode characters in patterns are handled. The user must ensure however that the URI passed to the Matcher methods, or to the --url parameter of the application, follows the format specified by RFC3986, since this library will not perform full normalization of those URI parameters. Only if the URI is in this format, the matching will be done according to the REP specification.

License

The robots.txt parser and matcher Java library is licensed under the terms of the Apache license. See LICENSE for more information.

Source Code Headers

Every file containing source code must include copyright and license information. This includes any JS/CSS files that you might be serving out to browsers. (This is to help well-intentioned people avoid accidental copying that doesn't comply with the license.)

Apache header:

Copyright 2020 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

It can be done easily by using the addlicense tool.

Install it:

$ go get -u github.com/google/addlicense

Use it like this to make sure all files have the licence:

$ ~/go/bin/addlicense -c "Google LLC" -l apache .