add robots.txt to Cilium documentation

[aanm]

It looks that we can't do a Page redirect from 'robots.txt' in
readthedocs settings. This commit is another attempt of having a
manually defined robots.txt file.

Signed-off-by: André Martins andre@cilium.io

[joestringer]

@aanm can you verify exactly which PRs need to be backported in order to backport this? I hit this issue during backporting, and it's not clear how to resolve it since the files are moving around and I'm unfamiliar with which PRs are doing what when it comes to the robots.txt:

$ cherry-pick  7d30daf013ab0dd86245bca64dd2d438d00d7dca
Applying: 7d30daf013ab add robots.txt to Cilium documentation                                                                  
error: mode change for Documentation/_static/sitemap-index.xml, which is not in current HEAD                                   
error: could not build fake ancestor                                                                                                                                                                                                                           
Patch failed at 0001 add robots.txt to Cilium documentation                                                                                                                                                                                                    
hint: Use 'git am --show-current-patch' to see the failed patch                                                                
When you have resolved this problem, run "git am --continue".                                                                                                                                                                                                  
If you prefer to skip this patch, run "git am --skip" instead.                                                                                                                                                                                                 
To restore the original branch and stop patching, run "git am --abort".

[aanm]

@joestringer this should be the final patch, you can probably get the same patch by doing $ cherry-pick 7d30daf013ab0dd86245bca64dd2d438d00d7dca && git add Documentation/robots/sitemap-index.xml && git cherry-pick --continue

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 6d02febb5d..0aa37c38d1 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -187,6 +187,11 @@ html_theme_options = {
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['images', '_static']
 
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+html_extra_path = ['robots']
+
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
diff --git a/Documentation/robots/robots.txt b/Documentation/robots/robots.txt
new file mode 100644
index 0000000000..e549b489e9
--- /dev/null
+++ b/Documentation/robots/robots.txt
@@ -0,0 +1,9 @@
+User-agent: *
+
+Disallow: /
+
+Allow: /en/stable
+
+Allow: /en/latest
+
+Sitemap: https://docs.cilium.io/en/latest/sitemap-index.xml
diff --git a/Documentation/robots/sitemap-index.xml b/Documentation/robots/sitemap-index.xml
new file mode 100644
index 0000000000..f040d1ac14
--- /dev/null
+++ b/Documentation/robots/sitemap-index.xml
@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
+        xmlns:xhtml="http://www.w3.org/1999/xhtml">
+    <url>
+        <loc>https://docs.cilium.io/en/stable/</loc>
+        <changefreq>daily</changefreq>
+        <priority>1</priority>
+    </url>
+    <url>
+        <loc>https://docs.cilium.io/en/latest/</loc>
+        <changefreq>daily</changefreq>
+        <priority>0.9</priority>
+    </url>
+</urlset>