Configure web scraper settings

garethbjohnson · Jun 30, 2018 · 60855b0 · 60855b0
1 parent 77f0a8b
commit 60855b0
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/static/robots.txt b/static/robots.txt
@@ -1,2 +1,2 @@
 User-agent: *
-Disallow: /
+Disallow: /books/