An App for testing URLs and User-agents blocked by robots.txt Files.
Given a certain robots.txt file, check whether or not any of the provided URLs can be fetched by all the user-agents in the robots.txt file (checks all the combinations).
-
Notifications
You must be signed in to change notification settings - Fork 0
Visual App for Testing URLs and User-agents blocked by robots.txt Files
eliasdabbas/robotstxt_app
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Visual App for Testing URLs and User-agents blocked by robots.txt Files
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published