Robustness and Networking

Robustness & Networking

Custom User-Agent

--user-agent sets the User-Agent header for every request. This matters more than it might seem: the default header sent by the underlying requests library (python-requests/x.y.z) is blocked outright by a lot of sites' bot-filtering. MailGrab ships its own reasonable default (Mozilla/5.0 (compatible; MailGrab/2.0; +https://github.com/oceanofanything/MailGrab)) and lets you override it entirely.

Retry with backoff

Every request goes through an HTTPAdapter with retry logic attached, but only for HTTP status codes — 500, 502, 503, and 504 responses are retried up to 3 times with exponential backoff. Connection and read failures (including timeouts) are deliberately not retried by this mechanism (connect=0, read=0 in the retry configuration) — retrying those would multiply the wait on a genuinely hung or unreachable server, defeating the entire point of --timeout.

The backoff delay between retries scales with your configured --timeout (max(0.1, min(timeout / 20, 2.0)) seconds) rather than being a fixed value — so a low timeout, chosen specifically to bound how long you're willing to wait per request, doesn't get quietly undermined by several seconds of fixed retry backoff on a flaky 5xx target.

Proxy support

--proxy <url> routes every request through a proxy. Supported schemes: http://, https://, socks4://, socks4a://, socks5://, socks5h:// — the SOCKS variants work via the PySocks package, which is a MailGrab dependency but otherwise sits unused. socks5h/socks4a (the "resolve DNS through the proxy" variants, the usual recommendation for something like a Tor SOCKS proxy) are supported, not just the base schemes.

The proxy URL is validated at startup — a malformed value (wrong scheme, missing host) exits immediately with a clear error, rather than failing silently and repeatedly on every single request once the crawl is already underway.

Timeout vs. connection-error logging

This one is subtle enough to be worth explaining directly, because it took two rounds to get right.

Mounting retry logic on the session (above) has a side effect: when a request genuinely times out, the underlying libraries (urllib3/requests) wrap it in a way that makes it surface as a generic ConnectionError instead of a proper Timeout exception. Left alone, that means every timeout gets logged as "Connection Error" — technically not wrong, but confusing when you're trying to tell "this site is just slow" apart from "this site is actually down."

MailGrab unwraps the disguised exception to recover the real cause and logs "Timeout Error" when that's what actually happened. The first version of this fix was too broad: it turns out urllib3's NewConnectionError (raised on a refused connection) and its NameResolutionError subclass (raised on a DNS lookup failure — by far the most common real cause of ConnectionError while crawling arbitrary discovered links, e.g. dead or typo'd domains) are themselves subclasses of the timeout exception class in urllib3's hierarchy. The first fix's isinstance check matched those too, so it ended up mislabeling a dead domain as a timeout — inverting the whole point of the fix for the single most common case. This was confirmed empirically (a nonexistent domain reliably raises NameResolutionError; a genuine connect-timeout raises the bare parent exception, not that subclass) and the check now explicitly excludes them.

The takeaway if you're reading logs: "Timeout Error" means the request genuinely timed out; "Connection Error" covers everything else (refused, DNS failure, and other connection-level problems).