Skip to content

v2.0.0: the kit's own Google Maps engine, no Docker required

Choose a tag to compare

View all tags
@Mahanaicoach Mahanaicoach released this 01 Nov 15:15
v2.0.0
01e270f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The kit now runs its own Google Maps engine, written in Python's standard library and driving a headless Chrome. No Go binary, no Playwright download and no Docker required: python3 scripts/engine.py start runs it on macOS, Linux and Windows, and docker compose up -d still works. The REST API, the job body and the 36 CSV columns are the same as 1.x, so your scripts and files keep working.

Upgrading from 1.x

  • Run bash scripts/update.sh. With Docker it rebuilds and restarts the engine on the same port and removes 1.x's two containers. Without Docker, start the new engine with python3 scripts/engine.py start.
  • Your 1.x jobs and their CSVs appear in the new web UI. A job 1.x had not finished shows as failed, with its rows. The old volume is kept, read-only.
  • A few columns hold slightly different data (plus_code, link, status, reviews on request, owner left empty). The full list is in SETUP.md, upgrading from kit 1.x.

New

  • scripts/engine.py start|stop|status|restart|jobs|delete. status confirms the engine answering is the kit's own (X-Engine header) and names its browser. With no Chrome, Chromium, Edge or Brave installed, the first start downloads Chrome for Testing once (about 100 MB, pinned by SHA-256).
  • Headless runs get Google's full listing view, so review_count is filled without opening a window.
  • Jobs are resumable. A job cut short by a restart picks up after its last finished keyword. Rows stream in while it runs, and the poll line shows progress.
  • Proxies with logins, rotated by the engine past the ones that fail, plus --country and --lang for the market. A refused proxy login or a Google block is named in the job's error, not left as a silent empty file.
  • --reviews N adds review text on request. images holds up to 20 business photo URLs. ?format=json on a download returns the data as scraped.
  • Socials (Instagram, Facebook, LinkedIn, TikTok, YouTube, X) are on by default. Turn them off with --no-socials.
  • A beginner-first web UI. The What box offers 201 popular business types, then Google's 4,053 category names. The Where box offers about 168,000 real cities and towns from GeoNames (CC BY 4.0), and each city runs as its own job centred on it.
  • The Claude skill only runs scripts/scrape.py: no curl, poll loop or eval in the skill text, and scraped values are treated as data, never instructions.

Fixed and hardened

  • scripts/scrape.py exits 1 when a run is incomplete (a timeout, a blocked keyword, failed places) and keeps a partial CSV on Ctrl-C.
  • The engine refuses any request whose Host or Origin is not its own localhost page. An optional SCRAPER_API_KEY is checked as an X-API-Key header.
  • Website fetches skip private and local addresses and have one hard deadline each. One engine per data folder.
  • CSV cells a spreadsheet would run as a formula are escaped. Names cut mid-emoji, very long cells and odd 1.x jobs no longer fail a job or the job list.
  • Deduplication keeps two different places that share a name. --out accepts only .csv or .json.

CI

  • Offline suites (scrape 93, engine 102, CDP 21 tests) on Linux, macOS and Windows, across every Python from 3.9 to 3.14.
  • A fresh-install job (docker compose up, health check, web UI, host and origin checks), a Docker build job, and live scrapes on four runners.
  • Before release, a real 1.x Docker install was upgraded with update.sh and checked end to end: jobs imported with their rows, and new scrapes ran.

Thanks

Assets 2
Loading