Ops and Resilience

Ops & Resilience

--append

By default, every run of MailGrab overwrites its output files fresh. --append changes that: before writing, it loads _results.json from a previous run (if one exists) and unions its emails, visited URLs, and source mappings into the current run's results, so the saved files accumulate across runs instead of replacing each other.

Note what this does and doesn't do: the crawl itself still starts fresh from the seed and re-fetches everything, exactly as if --append weren't set — the merging only happens at save time, after the crawl finishes. If you also want the crawl itself to skip pages it's already visited, see --resume below.

If _results.json exists but can't be parsed (corrupted, truncated, hand-edited into invalid JSON), MailGrab prints a visible warning and continues with just the current run's results, rather than silently discarding your accumulated history with no indication anything went wrong.

--resume

--resume does everything --append does, plus one more thing: it loads the previous _results.json before crawling starts, not just before saving. That means already-visited URLs are skipped during the crawl itself — they're never re-fetched at all — rather than being fetched again and only deduplicated afterward.

One thing worth understanding clearly: the seed URL itself is always re-fetched, even under --resume, even if it was already visited last time. This is intentional, not an oversight — MailGrab doesn't persist the actual link graph it discovered (which page links to which), only the flat list of what it visited and found. Re-fetching the seed is the only way to rediscover its current links at all; without that, --resume would have no way to find anything new. Every other already-visited page is correctly skipped.

This is a deliberately simpler design than full crawl-frontier persistence (saving the exact in-progress queue and picking up mid-traversal). It gets you the two things that matter most in practice — not re-fetching the bulk of what you already have, and not losing what you already found — without the complexity of a separate persisted-queue file. If you need to resume deep into a large interrupted crawl without re-walking any part of the graph from the seed, that's a reasonable next step to build on top of this.

# First run
python MailGrab.py --url https://example.com --depth 200

# Later, pick up where you left off
python MailGrab.py --url https://example.com --depth 200 --resume

--quiet

Suppresses all of MailGrab's normal output — the banner, the per-page "Processing:" lines, the progress updates, all of it — using Rich's own quiet mode on the console object. In its place, exactly one line is printed at the end: a JSON object with the final email count, URL count, and the list of output files written.

python MailGrab.py --url https://example.com --depth 50 --quiet

{"emailCount": 12, "urlCount": 48, "outputFiles": ["_emails.txt", "_scrappedUrls.txt", "_emails.csv", "_results.json"]}

This is meant for piping into another tool or a script that just wants a machine-readable result, not a scrolling console transcript. Fatal startup errors (an invalid --proxy, an out-of-range --depth) are still printed to stderr even under --quiet, specifically so a CI job using --quiet for clean output doesn't get a bare, unexplained non-zero exit code with nothing to explain it.