Webcrawler will enter your domain and scan every page of your website, extracting page titles, descriptions, keywords, and links etc..
install gskrawler using pip3 :
sudo pip3 install gskrawler
Usage:
visit this url : https://pypi.python.org/pypi/gskrawler