Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
35 lines (22 sloc) 1.19 KB

Can I Crawl (this URL)

Hosted robots.txt permissions verifier.


  • / This page.
  • /check Runs the robots.txt verification check.


Verifies if the provided URL is allowed to be crawled by your User-Agent. Pass in the destination URL and the service will download, parse and check the robots.txt file for permissions. If you're allowed to continue, it will issue a 3XX redirect, otherwise a 4XX code is returned.


$ curl -v

< HTTP/1.0 302 Found
< Location:

$ curl -v

< HTTP/1.0 403 Forbidden
< Content-Length: 23

$ curl -H'User-Agent: MyCustomAgent' -v

> User-Agent: MyCustomAgent
< HTTP/1.0 302 Found
< Location:

Note: disallows requests to /search.


MIT License - Copyright (c) 2011 Ilya Grigorik