Given a domain name, this tool will:
- Discover subdomains
- Look for a robots.txt file on each subdomain
- Generate some analytics on the contents of the robots.txt files
- In the
scripts
directory, copy thebotalyzer.example.env
tobotalyzer.env
- Edit this file with the settings for your preferred DNS service
./scripts/analyze.sh <domain>
First build the docker image:
npm run docker-build
Then run the docker container:
npm run docker-run
To generate the JSON summary for a domain, request the URL http://127.0.0.1:3000/example.com
For example:
curl http://127.0.0.1:3000/example.com