Block search engines with robots.txt to make it more difficult to for public instances to be abused

[pirate]

Right now it's pretty easy to use Google/Bing/DuckDuckGo to find public unsecured Send instances.

This is a security hazard to the server owners, as I'm sure you know the original Mozilla was shut down due to malicious users using it to host malware/CSAM.

I propose limiting search engine access to make it harder to discover these instances, as this will greatly limit the abuse/DMCA violations that public hosters have to deal with.

Advanced users wanting to allow search engine indexing will no doubt be running a reverse proxy in front of Send, which they can use to override the /robots.txt URL and return whatever content they want. This edit just makes it so that the bare, unsecured Send backend limits search engine access by default.

[timvisee]

This is a security hazard to the server owners, as I'm sure you know the original Mozilla was shut down due to malicious users using it to host malware/CSAM.

I have not seen any malicious activity for over a year on public instances (other than Mozilla's). I am assuming the bigger problem was that a big brand name that people trust was attached.

But I do agree that adding this is a good addition. Thanks!