Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
-
Updated
Mar 9, 2026 - HTML
Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
Visual robots.txt manager for the AI era. Block or allow GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers with one click.
Add a description, image, and links to the perplexitybot topic page so that developers can more easily learn about it.
To associate your repository with the perplexitybot topic, visit your repo's landing page and select "manage topics."