Skip to content
read robots.txt files from a list of sites and write them out
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md
talkToRobots.py

README.md

talkToRobots

reads robots.txt files then print and log what they say.

python3 talkToRobots.py -s https://github.com

python3 talkToRobots.py -f /path/to/file/of/sites

To Do

  • check for https if only http is provided and vice versa

  • option for disallow only

  • parse disallows to something...

You can’t perform that action at this time.