Create robots.txt

Disallow well-behaved robots from poking around in most places.

public/robots.txt

@@ -0,0 +1,17 @@
+# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
+
+User-agent: GPTBot
+Disallow: /
+
+User-agent: *
+Disallow: /api/inbox
+Disallow: /admin
+Disallow: /bookmark/
+Disallow: /comment
+Disallow: /login
+Disallow: /logout
+Disallow: /m/
+Disallow: /network
+Disallow: /search
+Disallow: /tagged/
+Disallow: /u/