hgwsgi: disallow crawling hg.mozilla.org (Bug 1830926) r=glob

Crawling on hg.mozilla.org has become a more and more frequent
problem in recent memory. This commit changes the `robots.txt`
file to be fully restrictive, except for the main landing
page and other pages which only list the existing repositories
on hg.mozilla.org.

Differential Revision: https://phabricator.services.mozilla.com/D176928

--HG--
extra : moz-landing-system : lando

hgwsgi/robots.txt

@@ -1,9 +1,14 @@
 User-agent: *
+Allow: /$
+Allow: /automation$
+Allow: /build$
+Allow: /ci$
+Allow: /conduit-testing$
+Allow: /hgcustom$
+Allow: /l10n$
+Allow: /l10n-central$
+Allow: /projects$
+Allow: /integration$
+Allow: /releases$
+Disallow: /
 
-Disallow: /*/annotate/
-Disallow: /*/graph
-Disallow: /*/graph/
-Disallow: /*/archive/
-Disallow: /*/diff/
-Disallow: /*/comparison/
-Disallow: /*/raw-*