Skip to content

v1.2.0: filters, proxies, email checks, reliability, CI

Choose a tag to compare

@Mahanaicoach Mahanaicoach released this 30 Oct 20:49
· 104 commits to master since this release
cf7678a

New

  • Row filters: --min-rating, --min-reviews, --no-website (businesses that still need a site), --has-phone, --has-email, --limit. --exclude earlier.csv skips the places you already have, so a weekly re-run returns only new leads.
  • --proxy URL, repeatable: the engine rotates them. socks5, socks5h, http, https.
  • --verify-emails: checks each email domain's MX record over DNS (no key), drops dead domains and adds email_host (google_workspace, microsoft_365, other).
  • --tech: Meta Pixel, Google Ads tag, analytics, Bing Ads, LinkedIn Insight and TikTok pixels, site platform, mobile support and a 0-100 intent_score that sorts the warmest leads first.
  • --socials also finds TikTok, YouTube and X. --lang sets the market's language. --city is repeatable: several cities into one deduped file.
  • Emails are now crawled by the script (homepage, then contact, about and team pages, 16 sites at a time): about 3x faster than the engine's email pass on the same listings, with more hits and no junk addresses.
  • Update with one command: bash scripts/update.sh (works in a clone or fork; your fork needs one Sync fork first to get the script).

Fixed

  • Engine runs 4 workers (-c 4): a depth 5 scrape went from about 8 minutes to about 1.
  • Web UI: the job list and Start Scraping button work again (gosom/google-maps-scraper#337) through the kit's nginx proxy, which also pins the API docs page's Redoc and sets the form to emails on, depth 5.
  • Engine hang after a finished job (gosom/google-maps-scraper#346): recovered automatically, including when an earlier web UI or API job left the engine stuck before your run; the stuck job is removed from the list. A failed job is retried once. The poll line shows rows as they arrive.
  • Grid tiles are no longer lost after an engine restart. Windows prints UTF-8. utm_ tracking stripped from websites. Junk and vendor emails filtered.
  • Security: the proxy refuses other hostnames (DNS rebinding) and cross-site posts; website fetches skip private and local addresses; the skill treats scraped data as untrusted.
  • Docs: the six required job fields (name, keywords, lang, lat, lon, max_time), skill-only install, troubleshooting for the engine hang.

CI

  • The offline test suite runs on ubuntu, macOS and Windows on every PR, plus a Linux fresh-install job (docker compose up, health check, patched web UI and API docs, proxy checks). No Google traffic.

Thanks

  • Community PRs that pointed the way: #11, #12, #13, #14, #15, #19, #20 (wonecosystem, jorgejuniordev and others). Engine: gosom/google-maps-scraper (MIT).