Permalink
Browse files

Merge pull request #523 from CristinaRO/Issues_1883_2847_robots

Issues 1883 and 2847: robots.txt: disallow indexing downloads and search results
  • Loading branch information...
2 parents b6f1b65 + b2e82d4 commit e4c63c5ce1ee1087ddedcfb58302d6ab0f9a3d86 @elzj elzj committed Mar 12, 2012
Showing with 10 additions and 0 deletions.
  1. +10 −0 public/robots.public.txt
View
@@ -7,11 +7,21 @@
User-agent: *
Disallow: /works? # cruel but efficient
Disallow: /autocomplete/
+Disallow: /downloads/
+Disallow: /external_works/
+# disallow indexing of search results
+Disallow: /bookmarks/search?
+Disallow: /people/search?
+Disallow: /tags/search?
+Disallow: /works/search?
User-agent: Googlebot
Disallow: /autocomplete/
+Disallow: /downloads/
+Disallow: /external_works/
# Googlebot is smart and knows pattern matching
Disallow: /works/*?
+Disallow: /*search?
Disallow: /*?*query=
Disallow: /*?*sort_
Disallow: /*?*selected_tags

0 comments on commit e4c63c5

Please sign in to comment.