Home

Robots.txt Generator

robots-generator writes a clean, valid robots.txt from a platform preset (WordPress, WooCommerce, Shopify, Magento, Next.js, Laravel or a locked-down staging site), your own Allow/Disallow rules and sitemap URLs. One flag adds a separate group that blocks 20 AI training and AI search crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider and more — and another blocks third-party SEO crawlers. Every result is linted for the mistakes that hurt SEO, and it fits straight into a build step.

• Getting started — install, first run, reading the output
• FAQ — common questions
• Troubleshooting — errors and how to fix them

Prefer a browser? The same check runs online at PromoPilot — free, no sign-up.