-
-
Notifications
You must be signed in to change notification settings - Fork 5
FAQ and Troubleshooting
At startup, MailGrab checks connectivity by pinging oceanofanything.github.io (via ping on Windows, curl on Linux/macOS) before doing anything else. If your network blocks outbound ICMP pings (common on corporate networks and some cloud/sandboxed environments) or can't resolve/reach that specific host, this check fails even though the rest of the internet is reachable fine. There's currently no flag to skip this check — if you're running MailGrab somewhere this check can't pass, you'll need to either allow that connectivity or patch the check locally.
If you just want to see the flag list, python MailGrab.py --help exits before the connectivity check ever runs — argparse's own --help handling short-circuits everything after it.
Because it was named before that distinction existed as a separate concept — see Architecture. Use --max-hops for link-distance from the seed; --depth remains the total-page-count cap.
This is a real limitation of Python's standard library, not a MailGrab bug: RobotFileParser.crawl_delay() only recognizes an integer delay value. Crawl-delay: 1 works; any fractional value is silently ignored. See Smarter Discovery.
This heuristic (turning word [at] word [dot] word into word@word.word) can occasionally match ordinary prose that happens to contain the literal words "at" and "dot" in that shape. It's a known, accepted tradeoff for catching genuinely obfuscated addresses — see Crawl Correctness.
Only a definitive "no such domain" or "no mail server" answer drops an email — a flaky/slow DNS lookup is supposed to keep it (see Email Quality). If you're confident an email was dropped incorrectly, check _MailGrabLog.txt for what the DNS lookup actually returned for that domain; also confirm the environment MailGrab is running in actually has working DNS resolution.
Check whether the site's robots.txt even has a Sitemap: line and whether /sitemap.xml exists at all — MailGrab only checks those two sources. If the sitemap is a <sitemapindex>, MailGrab resolves one level of it into child sitemaps automatically (see Smarter Discovery); a nested index (an index whose entries are themselves indexes, rather than page-listing sitemaps) is not currently followed further than that one level.
This only affects automated testing, not normal use: Python's logging.basicConfig() only takes effect on its first call in a process. If something runs MailGrab's script multiple times within one long-lived Python process (as test_mailgrab.py does, via runpy.run_path()), only the first run's log configuration actually applies — later runs' log output doesn't reliably land in a fresh _MailGrabLog.txt. A normal python MailGrab.py invocation is a fresh process every time and isn't affected.
Yes — --ignore-robots is required to disable that. See Crawl Correctness.
No, and there isn't meant to be a separate one — --depth (capped at 200 in batch mode, 500 in interactive mode by the input validation itself) already is the total-request safety valve. A second, independent cap was considered and skipped as redundant.
Open an issue on the GitHub repo. If you're proposing a new feature, check TODO.md in the repo first — there's a running list of known gaps and accepted tradeoffs.