Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

added robots.txt

  • Loading branch information...
commit 098092627448a9310c54b91e930c141535d63133 1 parent fbad153
@dano-reisys dano-reisys authored
Showing with 38 additions and 0 deletions.
  1. +38 −0 robots.txt
View
38 robots.txt
@@ -0,0 +1,38 @@
+# This is a demonstration staging server and should not be indexed by search engines.
+# The canonical site to crawl is http://data.gov
+
+User-agent: * # match all bots
+Disallow: / # keep them out
+
+User-Agent: Googlebot
+Disallow: /
+
+User-Agent: Googlebot-Mobile
+Disallow: /
+
+User-Agent: msnbot
+Disallow: /
+
+User-Agent: bingbot
+Disallow: /
+
+# Adsense
+User-Agent: Mediapartners-Google
+Disallow: /
+
+# Blekko
+User-Agent: ScoutJet
+Disallow: /
+
+User-Agent: Yandex
+Disallow: /
+
+User-agent: baiduspider
+Disallow: /
+
+User-agent: DuckDuckBot
+Disallow: /
+
+# CommonCrawl
+User-agent: ccbot
+Disallow: /
Please sign in to comment.
Something went wrong with that request. Please try again.