Update and rename robots.txt to robots.txt.dist

add instructions for allow/disalow robots
dotkernel · Apr 24, 2024 · 4c93c6b · 4c93c6b
1 parent a1c417e
commit 4c93c6b
Show file tree

Hide file tree

Showing 2 changed files with 9 additions and 2 deletions.
diff --git a/public/robots.txt b/public/robots.txt
diff --git a/public/robots.txt.dist b/public/robots.txt.dist
@@ -0,0 +1,9 @@
+# Below group disalow all robots.
+# To be used in DEVELOPMENT 
+User-Agent: *
+Disallow: /
+
+# Below group allow all robots.
+# To be used in PRODUCTION 
+User-Agent: *
+Allow: /