Repository navigation
v2.3.0: Leads CSV in the web UI, and more business emails
The web UI could only download the engine's raw CSV, so anyone who doesn't use a terminal had no way to get the clean outreach list that scripts/scrape.py writes (#48). This release adds that list to the web UI, gives the engine and the script one set of junk-email rules, and finds more business emails on their websites.
New
- Leads CSV in the web UI. Each job now has Leads CSV, Full CSV, JSON and Delete. Leads CSV gives the 8 lead fields (name, phone, email, website, category, address, rating, review count), then the social and
email_hostcolumns the job filled. It saves as<search>-leads.csv, and formulas are quoted like every other download. On the API it isGET /api/v1/jobs/{id}/download?format=leads. Thanks @themuzzamil for the clear write-up in #48. (#63) - One set of junk-email rules. The engine now drops the same addresses as the script: no-reply inboxes, tracking tokens, government addresses, and marketing or review vendors such as HubSpot, Mailchimp, SendGrid, Yext, Birdeye, Podium and Zocdoc. The Leads CSV applies them at download time, so jobs scraped before this release come out clean too. (#63)
Fixed: more business emails (scripts/scrape.py)
- Cloudflare-protected addresses are decoded. Many sites seemed to "have no email" only because Cloudflare hides it. (#64)
- Scheme and www retry. A homepage that fails as listed is tried again on the other scheme and with
www.toggled. Every retry keeps the existing guards (public-IP check, redirect guard, size cap, deadlines). A site that is down now costs up to 4 fetches instead of 1. (#64) - A social profile in the website slot fills that social column. When a listing's "website" is a Facebook, Instagram, LinkedIn, X, TikTok, YouTube or Yelp page, it is never crawled for emails, rendered or MX-checked. (#64)
- Deeper contact crawl. The crawl keeps going until it finds an address on the business's own domain, so a Gmail address on the homepage no longer ends the search. Contact-page link text now also matches location, impressum and kontakt pages. The same applies to the
--browser-sitesretry. (#64)
Docs
- The README comparison table uses check marks and adds five open-source rivals. (#62)
- The FAQ questions are now headings, with new answer-first questions. Added
AGENTS.mdfor Codex, Cursor and other agents, anddocs/best-google-maps-scrapers.md, a sourced comparison with other scrapers. (#36)
Tests
The offline suites pass on the merged code: scripts/test_scrape.py (113), engine/test_engine.py (102), engine/test_cdp.py (26) and engine/test_website_browser.py (20). Each PR also passed CI's live jobs on Linux, macOS, Windows, ARM and Docker. The merged build was not live-tested as a whole. Engine version 2.3.0.
Update with bash scripts/update.sh.