This repository has been archived by the owner on Apr 28, 2021. It is now read-only.
Validate robots.txt #3
Labels
Comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Some of the apps pages shouldn't be crawled. Apps generate a robots.txt for this purpose.
spidersuiteshould be able to validate the correctness of this file.The text was updated successfully, but these errors were encountered: