Add a 5-minute crawl delay to reduce crawl overhead

vipulnaik · Jul 29, 2020 · ca77301 · ca77301
1 parent 98b25b4
commit ca77301
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/access-portal/robots.txt b/access-portal/robots.txt
@@ -0,0 +1 @@
+Crawl-delay: 300