Performance and Concurrency

Performance & Concurrency

The original MailGrab fetched one page at a time, sequentially, with no connection reuse and no timeout — a single slow or hung server could stall the entire crawl indefinitely. This page covers how that was fixed.

Concurrent fetching

MailGrab crawls breadth-first: it fetches every page at the "current" BFS level, discovers their links, then moves to the next level. Within one BFS round, up to --concurrency (default 10) pages are fetched at the same time using a ThreadPoolExecutor, instead of one after another.

Concretely, each round:

1. Pops up to min(concurrency, depth remaining, urls queued) URLs off the frontier.
2. Submits all of them to the thread pool at once.
3. Waits for all of them to finish (via as_completed), collecting emails and newly-discovered links as each one comes back.
4. Repeats until the queue is empty or --depth's total-page budget is used up.

The very first round only ever has one URL (the seed), so it can't be parallelized — concurrency kicks in from the second round onward, once there's more than one URL to fetch.

Connection reuse

All requests share one requests.Session(), which pools and reuses the underlying TCP connections instead of opening a new one for every single page. The connection pool is sized to match --concurrency, so raising concurrency also gives you enough pooled connections to actually use it — an earlier version left the pool at its library default (10) regardless of how high you set concurrency, silently capping the benefit of anything above that.

Timeouts

Every request carries a timeout= (default 10 seconds, tune with --timeout). If a server hangs, that one request fails after the timeout and the crawl moves on — it no longer waits forever. See Robustness & Networking for exactly how a timeout is distinguished from other connection failures in the logs.

Rate limiting

--delay controls the minimum time between requests, but it's per-domain, not a single global pause — see Smarter Discovery for why that distinction matters and how robots.txt's Crawl-delay directive interacts with it.

Real progress reporting

Every completed BFS round prints Progress: X/depth Pages Crawled. An earlier version had a progress bar, but it only ever animated the file-saving step at the end of a run (100 fake iterations with no relation to how much crawling had actually happened) — it's gone now, replaced with this real per-round count.

Tuning tips

• Raising --concurrency helps most when crawling a single fast site with many pages to fetch — diminishing returns once you're limited by the target site's own response time rather than your own request rate.
• If you're crawling several different domains in one run (no --same-domain), the per-domain rate limit means concurrency across different domains isn't throttled by one domain's --delay — see Smarter Discovery.
• A very high --concurrency against a small site is indistinguishable from hammering it — see Crawl Correctness and use --delay if you're crawling something you don't want to overload.