Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Allow robots to roam around more in the /sites/ directory

The desire is to allow Google to index our uploaded files.
Also, allow an archiving service to make a copy of the
site, which requires access to our css and theming files.
  • Loading branch information...
commit c2a48708d607564147ee7a50502384838aad4db0 1 parent 90e15a1
@kwcoffman kwcoffman authored
Showing with 13 additions and 1 deletion.
  1. +13 −1 robots.txt
View
14 robots.txt
@@ -25,7 +25,19 @@ Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
-Disallow: /sites/
+# Here at U-M, we want to allow indexing of uploaded files
+# We also want to allow an archiving site access to our
+# theming and CSS
+# We want to continue to exclude the thumbnail directories ...
+Disallow: /sites/default/files/imagecache/
+Disallow: /sites/default/files/imagefield_thumbs/
+Disallow: /sites/default/files/imagefield_thumbssites/
+# ... and disallow access to our settings file(s)
+Disallow: /sites/default/settings.php
+Disallow: /sites/oer.switchback.com//settings.php
+# ... and continue to exclude libraries and modules
+Disallow: /sites/all/libraries/
+Disallow: /sites/all/modules/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Please sign in to comment.
Something went wrong with that request. Please try again.