Skip to content
robots.txt file generation based on a YAML config file
Ruby
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
lib
test
MIT-LICENSE
README
Rakefile
init.rb
install.rb
uninstall.rb

README

RobotsDotTxt
============

Installation:

- install as a plugin
- run rake robots:install
- add this route to your config/routes.rb file: match 'robots' => 'robots#index'
- edit the config/robots.yml file
- remove the public/robots.txt file



Example
=======

-- Example robots.yml file:

development:
  firstRule:
      userAgent:
          - google
          - some_other_bot
      allow: *
      disallow: new_post_path
      requestRate: 1/500s
      crawlDelay: 500
      visitTime: 0200-0800
  secondRule:
      userAgent: google-mobile
      allow: /mobile
      disallow: /xx      
      sitemap: /dev_sitemap.txt
  all:
      sitemap: /sitemap.xml



As you can see, you have to specify rules per each environment. You can use YAML lists, also Rails path names. Each rule has a name.


Copyright (c) 2011 novotarq, released under the MIT license
Something went wrong with that request. Please try again.