Robocop is a simple Rack middleware that inserts the X-Robots-Tag into the headers of all your responses
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Robocop Build Status

Robocop is a simple Rack middleware that inserts the X-Robots-Tag into the headers of all your responses.

The X-Robots-Tag can be used in place of a robots.txt file or meta tags to tell crawlers what they're allowed to do with your content. See this article for more information.

Tested against Ruby 1.8.7, 1.9.2, 1.9.3, and Rubinius & JRuby in both 1.8 and 1.9 mode.


The simplest way to install Robocop is to use Bundler.

Add Robocop to your Gemfile:

gem 'robocop'

Basic Usage


To use Robocop in your Rails application, add the following line to your application config file (config/application.rb for Rails 3, config/environment.rb for Rails 2):

config.middleware.use Robocop::Middleware do
  directives :all

Other Rack Applications (Sinatra, Padrino, etc.)

Simple add the following to your

use Robocop::Middleware do
  directives :all


The following directives can be passed in to Robocop's configuration:

  • all
  • noindex
  • nofollow
  • none
  • noarchive
  • nosnippet
  • noodp
  • notranslate
  • noimageindex

Directives (useragent agnostic)

If you just want to specify a list of directives for all useragents to follow, simply pass in a list of directives with the directive method:

config.middleware.use Robocop::Middleware do
  directives :noindex, :nofollow


If you want to give specific user agents unique sets of directives, you can do so by using the useragent method:

config.middleware.use Robocop::Middleware do
  useragent :googlebot do
    directive :all

  directives :noindex, :nofollow

It should be noted that if both the useragents and directives options are passed in, the useragents are output first in the header, followed by the generic directives.


  • Re-factor & DRY up code
  • Directive validation
  • Add support for unavailable_after directive.
  • Sanity checks for directives that are passed in. e.g. passing all, noindex, nofollow doesn't make any sense and should not be allowed.

Note on Patches / Pull Requests

  • Fork the project.
  • Code your feature addition or bug fix.
  • Add specs for it. This is important so we don't break it in a future version unintentionally.
  • Commit, do not mess with Rakefile or version number. If you want to have your own version, that's fine but bump version in a commit by itself so we can ignore it when merging.
  • Send a pull request. Bonus points for topic branches.


Copyright (c) 2012 PJ Kelly. See LICENSE for details.