Instead of crawling all the website, try to get urls from sitemap can use https://pypi.org/project/ultimate-sitemap-parser/
Instead of crawling all the website, try to get urls from sitemap
can use https://pypi.org/project/ultimate-sitemap-parser/