web spider control (#741)

* add setting to be added to article frontmatter to block web crawlers Signed-off-by: pcnorth <120174435+pcnorth@users.noreply.github.com> * remove sitewide whitelist from robot.txt Signed-off-by: pcnorth <120174435+pcnorth@users.noreply.github.com> --------- Signed-off-by: pcnorth <120174435+pcnorth@users.noreply.github.com> Signed-off-by: Phil North <120174435+pcnorth@users.noreply.github.com>
datatrails · Dec 14, 2023 · 8e93a0a · 8e93a0a
1 parent c81fb13
commit 8e93a0a
Showing 1 changed file with 0 additions and 4 deletions.
diff --git a/layouts/robots.txt b/layouts/robots.txt
@@ -1,8 +1,4 @@
 User-agent: *
-{{ if eq (hugo.Environment) "production" -}}
-Allow: /
-{{ else -}}
 {{ range where .Data.Pages "Params.robotsdisallow" true }}
 Disallow: {{ .RelPermalink }}{{ end }}
-{{ end }}
 Sitemap: {{ "sitemap.xml" | absURL -}}