Browse files

Allow robots to roam around more in the /sites/ directory

The desire is to allow Google to index our uploaded files.
Also, allow an archiving service to make a copy of the
site, which requires access to our css and theming files.
  • Loading branch information...
1 parent 90e15a1 commit c2a48708d607564147ee7a50502384838aad4db0 @kwcoffman kwcoffman committed Feb 22, 2011
Showing with 13 additions and 1 deletion.
  1. +13 −1 robots.txt
@@ -25,7 +25,19 @@ Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
-Disallow: /sites/
+# Here at U-M, we want to allow indexing of uploaded files
+# We also want to allow an archiving site access to our
+# theming and CSS
+# We want to continue to exclude the thumbnail directories ...
+Disallow: /sites/default/files/imagecache/
+Disallow: /sites/default/files/imagefield_thumbs/
+Disallow: /sites/default/files/imagefield_thumbssites/
+# ... and disallow access to our settings file(s)
+Disallow: /sites/default/settings.php
+Disallow: /sites/
+# ... and continue to exclude libraries and modules
+Disallow: /sites/all/libraries/
+Disallow: /sites/all/modules/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt

0 comments on commit c2a4870

Please sign in to comment.