Architecture

Architecture — What's Actually Happening Behind The Scenes

MailGrab is a single file, MailGrab.py. This page explains how it's actually built — the parts that matter if you're trying to understand why it behaves the way it does, or if you're extending it yourself.

Everything lives inside if __name__ == "__main__":

There's no importable module structure — every function, every class, every piece of config is defined inside that one top-level conditional block. This looks unusual, but it works the way you'd expect: if doesn't create a new Python scope, so functions defined inside it are still ordinary module-level functions underneath. That's why a helper function defined partway through the block can freely reference things like console, log, or a CLI-derived setting as if they were plain globals — because, mechanically, they are.

The practical upshot: you can't import anything from MailGrab.py from another script. The test suite (test_mailgrab.py) works around this by using runpy.run_path() to execute the whole script fresh each time, treating it as a black box driven through mocked input()/sys.argv/environment variables rather than calling functions directly.

Two entry points, one crawl engine

MailGrab can be started two ways — answer the interactive prompts, or give it a file of seed URLs (_inputUrls.txt, batch mode) — but both paths converge on exactly the same underlying functions:

• crawlUrls() — the BFS crawl loop itself.
• _fetchAndExtract() — fetches one URL and extracts emails/links/social-links from it.
• saveResults() — writes all four output files.

This wasn't always true. Early in MailGrab's history, batch mode and interactive mode each had their own ~250-line copy of the entire crawl loop, and their own ~70-line copy of the file-saving logic. One consequence of that duplication was a real, long-standing bug: batch mode's copy had its save/report/exit sequence accidentally nested one level too deep, inside a for loop over the scrapped URLs — meaning it silently re-ran on every single URL instead of once. Collapsing both entry points onto shared functions fixed that as a natural side effect of removing the duplication, not as a separate fix.

The BFS loop, concretely

crawlUrls()'s queue holds (url, hopCount) tuples, not bare URLs — that's what lets --max-hops cap link-distance independently of --depth's total-page-count cap (see Smarter Discovery). Each round:

1. Pops up to min(concurrency, depth remaining, queue length) items.
2. Filters out anything blocked by robots.txt (unless --ignore-robots).
3. Submits the rest to a ThreadPoolExecutor, applying the per-domain rate limit (see Smarter Discovery) before each submission.
4. Drains results via as_completed(), merging emails/links/social-links into the accumulator sets, and queuing newly-discovered links (respecting --same-domain, --max-hops, and the visited-set) for the next round.

Thread safety, or: why there are no locks anywhere

Worker threads (running _fetchAndExtract()) never touch shared mutable state directly — they only do I/O and pure computation, then return their findings. All merging of results into the shared visited/emails/emailSources/socialLinks/robotsCache accumulators happens back in the main thread, after as_completed() hands a result back. Because of that, none of MailGrab's concurrency needs a lock: there's exactly one thread ever writing to shared state, by construction, not by discipline.

The same accumulators are passed by reference into every seed's crawl in batch mode, which is why dedup state (and the robots.txt cache) is shared across all seed URLs in _inputUrls.txt, not just within one seed's own crawl.

The regex-backtracking bug, found twice

Two separate regexes in _fetchAndExtract() — the plain email-matching pattern and the [at]/[dot] de-obfuscation pattern — originally used unbounded quantifiers (+). Adversarial testing found that a page containing a long, unbroken run of word-like characters with no @/at/dot marker in it (a base64-encoded inline image, minified inline JavaScript, a long hash) triggered catastrophic regex backtracking: a few hundred kilobytes of such content could take minutes to hours to process on a single page, silently freezing the whole crawl with no error or timeout to catch it (the fetch itself had already succeeded — this was pure CPU-bound post-processing). Both regexes now use bounded quantifiers ({1,64} and similar, based on realistic maximum lengths for the parts of an email address) instead, which caps the backtracking cost per starting position to a constant and turns the whole thing back into roughly linear-time work. It's worth calling out specifically because it was fixed once, believed done, and then found again independently in the other regex during a second round of testing — a good reminder that "this pattern is safe" doesn't generalize from one regex to a sibling one that merely looks similar.

Other things adversarial testing caught along the way

MailGrab's feature set was built up over several rounds, each followed by independent adversarial review. A few of the more interesting findings, beyond the regex issue above:

• Batch mode losing everything on interruption. After the crawl-engine consolidation (above), the shared accumulators meant a KeyboardInterrupt or any other exception raised while crawling seed N would propagate out of the whole batch loop uncaught — discarding not just seed N's progress, but every seed before it too, since saveResults() was never reached. Batch mode now wraps its per-seed loop in the same try/except KeyboardInterrupt the single-URL path already had, so whatever was found before an interruption still gets saved.
• --url being silently ignored. If _inputUrls.txt existed, batch mode ran unconditionally and exited before the code that reads --url ever executed — so passing --url alongside an existing seed file crawled the file's URLs instead, with no warning that the flag was discarded. --url now explicitly takes priority over an existing seed file.
• Partial --url/--depth combinations hanging. Giving only one of the two flags (with no seed file to fall back on) would fall through to the other one's interactive prompt — harmless with a real terminal attached, but a silent hang (or an EOFError) in a script or CI job with no stdin to answer it. This now fails fast with a clear error instead.
• RobotFileParser.crawl_delay() silently returning None. The standard library's crawl_delay() only works if .modified() has been called on the parser first, which its own .read() method does automatically — but MailGrab deliberately avoids .read() (it makes an unbounded, timeout-free network request) and calls .parse() directly instead, which doesn't set that timestamp on its own. Confirmed by directly reading the standard library's source rather than assuming the API's behavior.

The pattern across nearly all of these: consolidating logic to remove duplication is almost always the right move, but every merge point (a shared accumulator, a shared cache, a shared exception handler) is exactly where a subtle bug likes to hide, and it's worth verifying each one explicitly rather than assuming the refactor is automatically safe.

Why "depth" caps total pages, not link-hops

This is a historical naming choice, not a bug: MailGrab's --depth has always meant "stop after this many total pages fetched," inherited from the very first version of the tool. --max-hops was added later specifically because "how many pages total" and "how many links deep" are genuinely different, useful controls, and by the time that need was clear, --depth already meant the former everywhere (the interactive prompt, both entry points, existing scripts). Renaming it would have been a breaking change for no functional benefit, so the two now coexist as independent, orthogonal caps.