[chore] Update robots.txt

[daenney]

Description

This updates the robots.txt based on the list of the ai.robots.txt repository. We can look at automating that at some point.

It's worth pointing out that some robots, namely the ones by Bytedance, are known to ignore robots.txt entirely.

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• I/we have read the GoToSocial contribution guidelines.
• I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
• I/we have not leveraged AI to create the proposed changes.
• I/we have performed a self-review of added code.
• I/we have written code that is legible and maintainable by others.
• I/we have commented the added code, particularly in hard-to-understand areas.
• I/we have made any necessary changes to documentation.
• I/we have added tests that cover new code.
• I/we have run tests and they pass locally with the changes.
• I/we have run go fmt ./... and golangci-lint run.