Home

MailGrab Wiki

MailGrab is a concurrent, robots.txt-aware email harvester: give it a URL (or a file full of them), it crawls the site's links, pulls out email addresses (including ones hidden behind obfuscation tricks), and saves everything in plain text, CSV, and JSON.

This wiki is the deep-dive reference. The README covers installation and a quick start; here you'll find every feature explained, how to use it, and what's actually happening behind the scenes.

Pages

Getting oriented

• Installation — getting MailGrab running on Windows, Linux, or macOS
• Quick Start — your first crawl, interactive and batch mode

Reference

• CLI Reference — every flag, what it does, its default
• Environment Variables — the MAILGRAB_* variables and how they relate to the flags
• Configuration File — --config, for saving a setup you reuse
• Output Files — _emails.txt, _results.json, and friends, with exact formats

Features, explained

• Performance & Concurrency — concurrent fetching, connection reuse, timeouts, rate limiting
• Crawl Correctness — same-domain scoping, URL normalization, mailto: parsing, de-obfuscation, robots.txt
• Smarter Discovery — sitemap seeding, Crawl-delay, hop limits, per-domain rate limiting
• Email Quality — case folding, placeholder filtering, MX validation
• Robustness & Networking — User-Agent, retries, proxies, and a subtle timeout-logging bug we had to fix twice
• Ops & Resilience — --append, --resume, --quiet

Under the hood

• Architecture — how the crawl engine is actually built, the concurrency model, and the bugs adversarial testing found along the way
• FAQ & Troubleshooting — common questions and known limitations

The 30-second version

# Interactive
python MailGrab.py

# Fully scripted, respecting the seed's own domain, with a custom User-Agent
python MailGrab.py --url https://example.com --depth 30 --same-domain --user-agent "MyBot/1.0"

# Resume a previous run, keeping what it already found
python MailGrab.py --url https://example.com --depth 30 --resume

Every crawl writes _emails.txt, _scrappedUrls.txt, _emails.csv, _results.json, and _MailGrabLog.txt to the current directory. See Output Files for what's in each one.