Sitemap generator that is scraping all links with puppeteer and generate sitemap.xml out of the data.
It validates all links, exclude links to files and broken pages.
clone the repo
cd into repo directory
run npm install
run npm start
You can parse the website in many cycles. Just restart the script. To generate new sitemap and clean all the data erase files in db/ directory.