Home

Robots.txt Checker

robots-checker downloads a site's robots.txt, parses it the way Google does (RFC 9309: user-agent groups, * and $ wildcards, longest-match wins, allow wins ties) and answers the question that matters: can this crawler fetch this URL, and which line decides it? It also flags the mistakes that quietly cost traffic — Disallow: / left over from staging, blocked CSS and JS, HTML served instead of a text file, rules before any User-agent, relative sitemap URLs and files over Google's 500 KB limit.

• Getting started — install, first run, reading the output
• FAQ — common questions
• Troubleshooting — errors and how to fix them

Prefer a browser? The same check runs online at PromoPilot — free, no sign-up.