Skip to content

MailGrab v2.0.0

Latest
Latest

Choose a tag to compare

View all tags
@OCEANOFANYTHING OCEANOFANYTHING released this 03 Sep 06:39
· 11 commits to main since this release
v2.0.0
5f4840f

The first tagged release. Version 2 reflects a major overhaul: MailGrab went from a single-threaded scraper to a concurrent, robots.txt-aware crawler with a new MCP server, structured output formats, and dozens of correctness/robustness fixes — all covered by a 43-test suite.

Highlights

New: MCP server for AI coding agents

mailgrab_mcp_server.py exposes MailGrab as a crawl_website tool over the Model Context Protocol, so Claude Code, Claude Desktop, Cursor, GitHub Copilot (VS Code and CLI), and Codex CLI can crawl a site and harvest emails directly instead of shelling out to the CLI by hand. Each call runs in an isolated subprocess/temp directory so concurrent tool calls can't corrupt the agent's stdio stream or clobber each other's output. See the wiki's MCP Server page for setup instructions per client.

Performance & concurrency

  • Concurrent fetching via ThreadPoolExecutor (--concurrency / MAILGRAB_MAX_WORKERS)
  • Shared requests.Session() with connection pooling and per-request timeouts (--timeout)
  • Configurable, per-domain rate limiting (--delay) that also respects robots.txt's Crawl-delay

Crawl correctness

  • --same-domain scoping and robots.txt compliance (--ignore-robots to opt out)
  • mailto: parsing as a direct email source; tel:/javascript:/anchor-only links skipped before fetching
  • De-obfuscation: name [at] domain [dot] com-style text and Cloudflare's data-cfemail spans
  • Real per-seed hop-limiting (--max-hops), independent of the total-page --depth cap
  • Sitemap-seeded discovery (--use-sitemap), including one level of <sitemapindex> resolution

Output

  • _emails.csv and _results.json (structured, with per-email source URLs and social/contact links) alongside the classic _emails.txt / _scrappedUrls.txt files
  • --append / --resume to build on a previous run instead of starting over
  • --quiet for a single JSON summary line, suitable for piping into other tools

Email quality

  • Case-insensitive dedup, placeholder-address filtering, and opt-in MX-record validation (--verify-mx)

Robustness

  • Custom --user-agent, proxy support (--proxy, including SOCKS), 5xx retry with backoff
  • Startup validation for --proxy / --depth with clear, non-zero-exit failures
  • Two catastrophic-backtracking (ReDoS) regexes found and fixed during adversarial testing

Documentation

  • Full GitHub Wiki rewrite: every feature, flag, and environment variable explained, plus an MCP Server setup guide verified against each client's current docs
  • README overhaul; dead documentation image links replaced

Testing

43 tests total: test_mailgrab.py (35, end-to-end against a local mock HTTP server) and test_mailgrab_mcp.py (8, driving the real MCP server through the official MCP client SDK).

For the complete, itemized rundown — including accepted edge cases and known limitations — see TODO.md and CLAUDE.md.

Assets 2
Loading