Adds robots.txt

closes #333
MITLibraries · Aug 24, 2015 · 20598da · 20598da · richardrodgers · Aug 24, 2015
1 parent 91817c3
commit 20598da
Show file tree

Hide file tree

Showing 2 changed files with 12 additions and 0 deletions.
diff --git a/conf/routes b/conf/routes
@@ -196,3 +196,4 @@ POST    /submodel                    controllers.Application.addSubscriberModel
 
 # Map static resources from the /public folder to the /assets URL path
 GET     /assets/*file               controllers.Assets.at(path="/public", file)
+GET     /robots.txt                 controllers.Assets.at(path="/public", file="robots.txt")
diff --git a/public/robots.txt b/public/robots.txt
@@ -0,0 +1,11 @@
+User-agent: *
+Disallow: /item
+Disallow: /item/
+Disallow: /items
+Disallow: /items/
+Disallow: /topic
+Disallow: /topic/
+Disallow: /topics
+Disallow: /topics/
+Disallow: /search
+Disallow: /search/