sitemapper

Site-recon tooling for scraper development. Given a base URL, sitemapper finds the site structure (robots.txt and sitemaps). It can also crawl internal pages and map internal and outgoing links into a graph.

Quickstart

pip install sitemapper
pip install sitemapper[stealth]   # recommended: adds curl_cffi TLS impersonation

from sitemapper import discover, crawl

# Passive: no HTML crawled. Fetches robots.txt and all sitemaps.
info = discover("https://www.python.org")
print(info.summary())
# Base URL:         https://www.python.org
# Sitemaps found:   1
# URLs in sitemaps: 342
# Crawl-delay:      None

# Active: bounded BFS that builds a link graph.
graph = crawl("https://www.python.org", max_pages=50, max_depth=2)
print(graph.summary())
# Pages crawled (internal): 50
# External URLs seen: 87
# Top external domains: docs.python.org (23), ...

Entry points

Function / class	Description
discover(url)	Passive discovery: robots.txt and sitemaps, returns a SiteDiscovery.
crawl(url)	Bounded BFS crawl, returns a LinkGraph.
Sitemapper	Stateful class. Holds config and transport, and caches discovery results.

Transport

All HTTP traffic goes through unblock_requests.CloudflareSession (env prefix SITEMAPPER). This lets recon work on Cloudflare-fronted sites.

export SITEMAPPER_FLARESOLVERR_URL=http://localhost:8191   # point at a running FlareSolverr
export SITEMAPPER_FLARESOLVERR_FALLBACK=1                  # escalate blocked GETs to solver
export SITEMAPPER_WAYBACK_FALLBACK=1                       # fall back to Wayback Machine

Or configure it in code:

from sitemapper import Sitemapper
sm = Sitemapper(flaresolverr_url="http://localhost:8191", wayback_fallback=True)
info = sm.discover("https://www.progarchives.com")

CLI

# Passive discovery (default)
python -m sitemapper https://www.python.org

# Active crawl with export
python -m sitemapper https://www.python.org \
    --crawl --max-pages 50 --json graph.json --dot graph.dot

# Cloudflare-fronted site
python -m sitemapper https://example.com --flaresolverr http://localhost:8191

Full flag list: python -m sitemapper --help

LinkGraph

crawl() returns a LinkGraph. It has no third-party dependencies and stores its data as a plain dict adjacency:

graph.nodes           # dict[str, Node]: url, status, title, depth, ...
graph.internal        # set[str]
graph.external        # set[str]
graph.domains()       # {host: count}  outgoing external links
graph.to_json()
graph.to_dot()        # Graphviz digraph (dot -Tsvg out.dot)
graph.summary()

Related projects

• unblock_requests supplies the transport layer that handles Cloudflare-fronted sites.
• anon_requests adds IP rotation, used by the optional sitemapper[anon] extra.

Docs

• docs/quickstart.md
• docs/discovery.md
• docs/crawl_graph.md
• docs/transport.md
• docs/cli.md
• docs/site-survey.md