Make robots.txt compliant with RFC (#1895)

The [RFC](http://www.robotstxt.org/orig.html) mentions only `Disallow` directive, so it must appear in the file. `Allow` is an ad hoc agreement between search engines that no all of them follow.
getzola · Jun 12, 2022 · d4b5c4f · d4b5c4f
1 parent c70aaf7
commit d4b5c4f
Show file tree

Hide file tree

Showing 3 changed files with 3 additions and 0 deletions.
diff --git a/components/templates/src/builtins/robots.txt b/components/templates/src/builtins/robots.txt
@@ -1,3 +1,4 @@
 User-agent: *
+Disallow:
 Allow: /
 Sitemap: {{ get_url(path="sitemap.xml") }}
diff --git a/docs/content/documentation/templates/robots.md b/docs/content/documentation/templates/robots.md
@@ -11,6 +11,7 @@ and the default is what most sites want:
 
 ```jinja2
 User-agent: *
+Disallow:
 Allow: /
 Sitemap: {{/* get_url(path="sitemap.xml") */}}
 ```
diff --git a/test_site/templates/robots.txt b/test_site/templates/robots.txt
@@ -1,3 +1,4 @@
 User-agent: zola
+Disallow:
 Allow: /
 Sitemap: {{config.base_url}}/sitemap.xml