-
-
Notifications
You must be signed in to change notification settings - Fork 5
Smarter Discovery
Features about finding the right pages and pacing requests sensibly, beyond plain link-following.
--use-sitemap adds an extra discovery source at the start of the crawl: MailGrab checks the seed's robots.txt for any Sitemap: line(s), and also always tries the conventional default path, /sitemap.xml. Every URL found this way is added to the crawl queue at hop 1 (see hop depth below), alongside whatever the seed page's own <a> links produce.
Sitemaps come in two shapes, and both are handled:
- A flat
<urlset>— its<loc>entries are page URLs, used directly. - A
<sitemapindex>— the layout WordPress (with Yoast/RankMath), Shopify, and most CMS defaults actually use. Its<loc>entries point to other sitemap files, not pages. MailGrab fetches those child sitemaps too (one level deep) and pulls the real page URLs out of them, rather than mistakenly treating a sitemap-index's entries as content pages (which was a real bug caught by adversarial testing — an earlier version got zero benefit from--use-sitemapon any site using this extremely common layout).
This is opt-in rather than default-on because it adds one to a few extra HTTP requests to every single crawl, whether or not you actually wanted the broader discovery.
There's no cap on how large a fetched sitemap can be — a huge or malicious sitemap could add real latency/memory use before --depth gets a chance to bound anything. Low real-world risk (not observed to cause problems in practice), but worth knowing about if you're pointing MailGrab at an untrusted site.
MailGrab has always had --depth, which — despite the name — caps the total number of pages fetched, not how many links deep the crawl follows from the seed. --max-hops is the independent control for that: --max-hops 2 means "only follow links that are at most 2 clicks from the seed," regardless of how high --depth is set. --max-hops 0 means "fetch only the seed itself, follow nothing."
The two combine naturally: --depth is still the hard ceiling on total requests either way; --max-hops additionally prunes the shape of what gets explored. Sitemap-seeded URLs (hop 1) respect --max-hops the same way regular discovered links do.
If a site's robots.txt specifies a Crawl-delay, MailGrab honors it — the effective delay for any given request is max(your --delay, that site's Crawl-delay), so a site asking for a slower crawl than you configured gets the slower one.
One real limitation, inherited from Python's standard library: RobotFileParser.crawl_delay() only recognizes an integer value. Crawl-delay: 1 works; Crawl-delay: 0.3 is silently ignored (treated as if no delay were specified at all). This was confirmed directly against the standard library's source during development — it's not something MailGrab can fix without writing its own robots.txt parser, which wasn't judged worth the complexity for this edge case.
--delay (and any robots.txt Crawl-delay) is enforced per domain, not as one global pause between every request MailGrab makes. Concretely: MailGrab tracks the last time it submitted a request to each domain, and only waits before a new request if that specific domain was hit too recently.
Why this matters: if you crawl without --same-domain and your seed's page links out to two other sites, a --delay 1 shouldn't mean "wait a full second between literally every single request across all three domains" — that would make a multi-domain crawl needlessly slow for no politeness benefit (the other two domains have never been touched, so there's nothing to rate-limit against yet). Per-domain tracking means each domain gets its own pacing, and domains that haven't been hit recently are never held up by one that has.