Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Added a few default disallows to robots.txt

  • Loading branch information...
commit 22cf150172ee5fe38f2e762ee707ee338ef301f3 1 parent 032757b
@jhermsmeier authored
Showing with 15 additions and 0 deletions.
  1. +15 −0 robots.txt
View
15 robots.txt
@@ -0,0 +1,15 @@
+
+User-agent: MSIECrawler
+Disallow: /
+
+User-agent: psbot
+Disallow: /
+
+User-agent: Fasterfox
+Disallow: /
+
+User-agent: Xenu
+Disallow: /
+
+User-agent: SiteSucker
+Disallow: /
Please sign in to comment.
Something went wrong with that request. Please try again.