-
-
Notifications
You must be signed in to change notification settings - Fork 5
Email Quality
Features that clean up what gets saved, not just what gets crawled.
Email addresses are effectively case-insensitive in practice (almost no real mail server treats User@Example.com and user@example.com as different mailboxes), so MailGrab lowercases every address the moment it's collected — from the plain-text regex, from mailto: links, and from de-obfuscated text alike — before adding it to the results.
This normalization is also re-applied when loading a previous run's _results.json for --append/--resume: if that file came from before this feature existed, or was hand-edited, or was produced by some other tool, its entries get lowercased on the way in too. Otherwise merging with an un-normalized file could silently reintroduce the exact case-duplicate problem this feature exists to solve — just moved from "within one run" to "across runs."
Template/boilerplate markup often contains a fake "example" email address that isn't a real contact — _isPlaceholderEmail() filters these out. When something is filtered, MailGrab says so on the console and in the log (Filtered N Placeholder Email(s): ...) — it's never a silent drop.
The filter has two parts, deliberately asymmetric:
-
An exact list of known placeholder addresses (
example@example.com,test@test.com,your-email@domain.com, and similar) — safe to block outright since these specific strings only ever show up as literal template filler. -
A short list of whole domains that can genuinely never have a real mailbox:
example.com,example.org, andexample.net(reserved by RFC 2606 specifically for documentation — no real mail server will ever answer for them), pluswixpress.com(Wix's own internal domain for auto-generated site-builder addresses, not something a real business uses as its actual contact).
That domain list is intentionally short. An earlier version also blocked domain.com, email.com, test.com, company.com, and sentry.io wholesale — all of which turned out to be real, live domains that genuinely hand out real mailboxes (email.com is an actual webmail provider; domain.com is a real registrar's own domain). Blocking a whole domain is a much bigger hammer than blocking one specific fake address, and adversarial testing caught real addresses being silently destroyed by that overreach. If you're ever tempted to add a domain to this list, make sure it genuinely can never have a real mailbox first — otherwise add the specific fake address to the exact-match list instead.
--verify-mx checks that each found email's domain actually has a mail server configured (an MX DNS record) before saving it, dropping anything that doesn't. This catches typo'd domains, expired domains, and other addresses that would simply bounce.
It's opt-in, for two reasons: it adds one DNS lookup per unique domain found (not per email — if ten emails share a domain, that's one lookup, done concurrently with the rest), and it requires working DNS resolution, which not every environment has.
Only a definitive answer drops an email. If the DNS lookup comes back NXDOMAIN (the domain doesn't exist) or NoAnswer (the domain exists but genuinely has no MX record), the email is dropped. Any other failure — a resolver timeout, no nameservers reachable, a flaky network — is treated as inconclusive, and the email is kept, with a warning printed instead. This distinction matters: an earlier version treated every DNS failure identically, which meant a slow or temporarily unreachable resolver could silently erase perfectly real, deliverable addresses with no way to tell the difference from a genuinely bad domain.