This was for my tutorial on building a dead link checker so its scope has been kept quite small.
Broken Link Crawler
Let's say I have a website and I want to find any dead links and images on this website.
$ python deadseeker.py 'https://healeycodes.com/' > 404 - https://docs.python.org/3/library/missing.html > 404 - https://github.com/microsoft/solitare2
It's that simple. The website is crawled, and all
src attributes are sent a request. Errors are reported. This bot doesn't observe
robots.txt but you should.
It is not a clever bot. But it is a good bot.
Accepting (small) PRs and issues!