Apparently robots.txt is not enough if someone links to a page.

stefansundin · Nov 13, 2018 · 6bcc1eb · 6bcc1eb
1 parent 6a6976a
commit 6bcc1eb
Show file tree

Hide file tree

Showing 2 changed files with 4 additions and 1 deletion.
diff --git a/app.rb b/app.rb
@@ -8,6 +8,10 @@
 
 before do
   content_type :text
+  # only allow search engine indexing of the main page
+  if request.path != "/"
+    headers({"X-Robots-Tag" => "noindex"})
+  end
 end
 
 get "/" do

diff --git a/public/robots.txt b/public/robots.txt
@@ -1,4 +1,3 @@
 User-agent: *
 Allow: /$
-Allow: /live$
 Disallow: /