Better robots.txt

Removed crawl-delay as non respected and added exclusion for all modules directory except theme
Ortus-Solutions · Mar 3, 2017 · 9fb4031 · 9fb4031
1 parent 34536b4
commit 9fb4031
Showing 1 changed file with 3 additions and 3 deletions.
diff --git a/robots.txt b/robots.txt
@@ -1,6 +1,6 @@
-User-agent: *
+User-agent: *
 Disallow: 
-Crawl-delay: 60
+Allow: /modules/contentbox/themes/
 Disallow: /coldbox/
 Disallow: /config/
 Disallow: /handlers/
@@ -11,4 +11,4 @@ Disallow: /logs/
 Disallow: /model/
 Disallow: /modules/
 Disallow: /plugins/
-Disallow: /views/
+Disallow: /views/