Skip to content

v2.1.0: optional browser rendering for website emails and social links

Choose a tag to compare

View all tags
@Mahanaicoach Mahanaicoach released this 01 Nov 18:16
v2.1.0
334c2b4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Some business websites build their contact details with JavaScript, so a plain HTTP fetch sees no email address or social links. The kit can now open those sites in a real browser. This is optional and off by default. Without it, the default crawl works exactly as in 2.0.0, with no pip install.

New

  • --browser-sites (CLI), "browser_sites": true (API) and Render websites (Playwright) (web UI). When the quick HTTP check of a business website finds no email or social profile links, the kit opens the site in headless Chrome and reads the rendered page. For emails, it also opens the contact and about pages.
  • Setup: install Playwright in a virtual environment with requirements-browser.txt, then run the scripts or start the engine with that environment's Python. The SETUP.md section has the commands. It uses the Chrome the kit already finds, with no extra browser download.
  • A new website_fetch_status column shows how each site was read: http, rendered, blocked, timeout, budget_exhausted or error.
  • The web UI has help notes for Social accounts and Render websites that say what each option collects and how it affects speed.

Speed and limits

  • All HTTP lookups finish first. Browser retries then share one browser per job, two pages at a time, within 30 seconds. In a live test, 20 Austin plumbers took 36 s with retries and 31 s without them, and emails were found for 8 businesses instead of 6.
  • Logins, CAPTCHAs and access-denied pages are not solved. Those sites report blocked.
  • The stock Docker image does not include Playwright. There the option returns an error that points to the setup steps.

Safety

  • Browser pages go through a local proxy that allows public IP addresses only. Redirects, scripts and iframes cannot reach your router or other local addresses. Images, media and fonts are not downloaded.
  • Each page gets a fresh browser context, and the browser is always shut down when the job ends. A graceful close is capped at 5 seconds.

Update with bash scripts/update.sh.

Assets 2
Loading