First pass at blocking googleweblight crawler

🐿 v2.8.0
Financial-Times · Apr 16, 2018 · 5aa0d32 · 5aa0d32
1 parent 4975d80
commit 5aa0d32
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/src/assets/robots.txt b/src/assets/robots.txt
@@ -1,5 +1,8 @@
 User-agent: Twitterbot
 Disallow:
+
+User-agent: googleweblight
+Disallow:
 
 User-agent: *
 Disallow: /