Try harder to allow archive.org to index old specs

See http://webmasters.stackexchange.com/q/71377/11849. Hopefully fixes #19 better.
whatwg · Feb 16, 2017 · e4e8b92 · e4e8b92
1 parent 1742f99
commit e4e8b92
Showing 1 changed file with 6 additions and 0 deletions.
diff --git a/src/robots.txt b/src/robots.txt
@@ -1,6 +1,12 @@
 User-agent: archive.org_bot
 Disallow:
 
+User-agent: ia_archiver
+Disallow:
+
+User-agent: ia_archiver-web.archive.org
+Disallow:
+
 User-agent: *
 Disallow: /specs/url/
 Disallow: /specs/web-apps/