-
-
Notifications
You must be signed in to change notification settings - Fork 5
Crawl Correctness
Features that make the crawl itself more accurate — following the right links, skipping the wrong ones, and finding emails that a plain regex over the visible text would miss.
--same-domain drops any discovered link whose domain (netloc) doesn't match the seed URL's, before it's ever queued. Without this flag, MailGrab will happily follow an outbound link to a completely different site — sometimes exactly what you want (mapping out a small web of related pages), sometimes not (you asked it to scan one company's site and it wandered off into a footer link to a social network, a partner's site, or beyond).
Every discovered link has its #fragment stripped (https://example.com/page#section → https://example.com/page) before being compared against the set of already-visited URLs. Without this, the same page reached with and without a fragment would be treated as two different URLs and fetched twice. The seed URL itself is normalized the same way.
Query-string normalization (treating ?a=1&b=2 and ?b=2&a=1 as the same URL) is not implemented — it was judged unnecessary complexity for the actual problem (fragments are overwhelmingly the common source of duplicate-looking URLs; query-parameter reordering is rare in the wild). If it becomes a real problem for your use case, that's a reasonable, small addition.
Links ending in a known non-page extension are never fetched at all:
Images: .png .jpg .jpeg .gif .svg .webp .ico .bmp
Archives/docs: .pdf .zip .rar .7z .gz .tar
Code/style: .css .js .mjs .map
Media: .mp3 .mp4 .avi .mov .wav .ogg
Fonts: .woff .woff2 .ttf .eot
Data: .xml
The check looks at the URL's path, not the full URL with its query string — so /photo.jpg?size=large is correctly skipped, not just a bare /photo.jpg.
tel:, javascript:, and bare # anchors are also never queued as crawl targets (checked case-insensitively, so TEL:... is caught the same as tel:...).
A mailto: link is a much more reliable email source than scanning visible page text — the address is already unambiguous and exact. MailGrab pulls the address straight out of mailto: hrefs (stripping any ?subject=... query the link might carry) in addition to its regex-based text scanning, case-insensitively (MAILTO:... is recognized the same as mailto:...).
Two techniques catch emails that sites deliberately hide from simple scrapers:
Cloudflare's "Email Address Obfuscation" rewrites a real address into <span class="__cf_email__" data-cfemail="...">, where the value is the address XOR-encoded with a key byte. MailGrab decodes it back to the real address.
Text-based obfuscation like person [at] example [dot] com or person (at) example (dot) com is matched with a dedicated regex and rewritten to person@example.com. This one is a heuristic — it can occasionally produce a false positive if ordinary prose happens to contain a "word at word dot word" pattern that isn't actually an email (rare, but possible). Both regexes involved use bounded quantifiers rather than unbounded ones — an earlier version's unbounded pattern could take minutes to hours to process a single page containing a long unbroken run of text with no @/at/dot marker (a base64-encoded image, a minified inline script), a classic regex-backtracking trap found through adversarial testing. See Architecture for the full story.
By default, MailGrab checks robots.txt before fetching any URL and skips pages it's told not to crawl (--ignore-robots disables this). The check is fetched through the same shared session and timeout as everything else — deliberately not using Python's built-in RobotFileParser.read(), which makes its own unbounded network request with no timeout of its own. A robots.txt that's missing, unreachable, or returns an error is treated as "allow everything," and the result is cached per-domain so it's only fetched once even across multiple seed URLs in a batch run.
robots.txt's Crawl-delay directive is also read and enforced — see Smarter Discovery.
Links to LinkedIn, Twitter/X, Facebook, or Instagram, and links whose path looks like a contact page (contact, contact-us, get-in-touch), are recorded separately in _results.json's socialLinks array instead of being queued and crawled as ordinary pages — they usually sit right next to the emails you're actually after, so it's a useful bonus without extra requests.