bskyweb: add robots disallow and 'canonical' config flags

[bnewbold]

NOTE: it is critical to enable the "canonical" flag for our prod deployment before merging/deploying this branch.

Adds two config/envvar flags:

• one to allow swapping in a blank "Disallow: *" robots.txt to prevent crawling (eg, in dev/staging/informal deployments)
• another to only serve some endpoints in the "canonical" deployment (eg, bsky.app)

Normally we might try to detect what "environment" we are in and change these behaviors based on that, but we have kind of a sprawling menagerie of environments, and folks also might be forking this repo for their own use, so made these more granular flags. Open to changing path and doing this the traditional ENVIRONMENT variable way though.

[render]

Your Render PR Server URL is https://social-app-pr-4760.onrender.com.

Follow its progress at https://dashboard.render.com/web/srv-cq6r43mehbks73bfl4ug.

[github-actions]

Old size	New size	Diff
6.61 MB	6.61 MB	0 B (0.00%)

---

[gaearon]

lgtm but i don't go