Website sitemap parser for Python 3.5+.
- Supports all sitemap formats:
- Field-tested with ~1 million URLs as part of the Media Cloud project
- Error-tolerant with more common sitemap bugs
- Tries to find sitemaps not listed in
- Uses fast and memory efficient Expat XML parsing
- Doesn't consume much memory even with massive sitemap hierarchies
- Provides a generated sitemap tree as easy to use object tree
- Supports using a custom web client
- Uses a small number of actively maintained third-party modules
- Reasonably tested
pip install ultimate-sitemap-parser
from usp.tree import sitemap_tree_for_homepage tree = sitemap_tree_for_homepage('https://www.nytimes.com/') print(tree)
sitemap_tree_for_homepage() will return a tree of
AbstractSitemap subclass objects that represent the sitemap
hierarchy found on the website; see a reference of AbstractSitemap subclasses.
If you'd like to just list all the pages found in all of the sitemaps within the website, consider using
# all_pages() returns an Iterator for page in tree.all_pages(): print(page)
all_pages() method will return an iterator yielding
SitemapPage objects; see a reference of SitemapPage.