Permalink
Browse files

Added a few default disallows to robots.txt

  • Loading branch information...
1 parent 032757b commit 22cf150172ee5fe38f2e762ee707ee338ef301f3 @jhermsmeier committed Apr 27, 2012
Showing with 15 additions and 0 deletions.
  1. +15 −0 robots.txt
View
@@ -0,0 +1,15 @@
+
+User-agent: MSIECrawler
+Disallow: /
+
+User-agent: psbot
+Disallow: /
+
+User-agent: Fasterfox
+Disallow: /
+
+User-agent: Xenu
+Disallow: /
+
+User-agent: SiteSucker
+Disallow: /

0 comments on commit 22cf150

Please sign in to comment.