Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Removing robots.txt from public #757

Merged
merged 1 commit into from

2 participants

@jasonfox

I recently ran into an issue where I was trying to dynamically serve robots.txt from Rails, i.e., map "/robots.txt", => "welcome#robots". The reason for this is that I want to serve one version of the file in staging and a different version of the file in production. So, I implemented this approach and removed the default Rails robots.txt from my application's public folder only to find that robots.txt was still being picked up from somewhere. I determined that the culprit was the rails_admin gem.

I've searched for a way to prevent the robots.txt file from being picked up without having to modify rails_admin but have come up short. Since the robots.txt file in the gem serves no purpose and since it is conflicting with Rails it doesn't seem that it would hurt to remove it. However, let me know if you disagree or if you know of a different way to resolve this issue.

Thanks,
Jason

@jasonfox jasonfox Removing robots.txt from public as it seems to get picked up if you r…
…emove robots.txt from your Rails app which prevents you from being able to serve robots.txt using a Rails controller action
ac1646f
@sferik sferik merged commit 1d1622a into sferik:rails-3.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 5, 2011
  1. @jasonfox

    Removing robots.txt from public as it seems to get picked up if you r…

    jasonfox authored
    …emove robots.txt from your Rails app which prevents you from being able to serve robots.txt using a Rails controller action
This page is out of date. Refresh to see the latest.
Showing with 0 additions and 5 deletions.
  1. +0 −5 public/robots.txt
View
5 public/robots.txt
@@ -1,5 +0,0 @@
-# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
-#
-# To ban all spiders from the entire site uncomment the next two lines:
-# User-Agent: *
-# Disallow: /
Something went wrong with that request. Please try again.