Respect robots.txt

[ineffyble]

If this tool is going to be trying to crawl my instance, I'd like it to respect robots.txt.

[tedivm]

The update with robots.txt support was just deployed- thanks for opening the ticket!

[ineffyble]

I'd recently made my domain_blocks return a HellPot, so I figured it'd be advantageous for you too. Thanks for responding so quickly!

[tedivm]

No problem- and sorry for not checking the robots file right away! I added the user agent yesterday to make sure people could find me if the bot caused problems, and it's only been up for about two days now. I'm really hoping to turn this into a tool that makes it easier for people to avoid joining hostile/bigoted servers and maybe even makes it easier for admins to block those servers, but it's still early days for the project. If you have any other feedback or ideas I'm super happy to hear them!