This is a python crawler that disregards robots.txt rules and downloads disallowed resources
Install dependencies:
make installRun:
python badrobot.py <domain>Example:
python badrobot.py https://www.example.comPlease note that the projet uses python 3, so on some systems you might need to use
python3instead ofpython.