v1.0.0 — BBB scraper, rebuilt

First release of the rebuilt scraper. It replaces four standalone scripts that predated the template — no shared row model, no exit-code contract, no sidecar, no tests.

Read this first

--mode search and --mode category need no account, no key and no proxy.

BBB renders its listings out of its own JSON endpoint, and that endpoint is not behind Cloudflare. Measured 2026-09-16 from a datacenter VPS — an address that gets HTTP 403 on every HTML page on the site — the endpoint answered HTTP 200 with 15 complete business records. A two-page run from that machine, with no credentials configured at all, returned 30 rows and status: complete.

--mode profile is the one that needs a residential exit, and it is what the paid products are for here: a profile page is gated, and there is no endpoint behind it.

What's in it

• Three engines — Playwright (primary, the only one with --concurrency), Selenium, pyppeteer — plus a browserless client for 2Captcha's Scraper API. All produce identical rows and exit codes.
• Three modes: a keyword search, a category listing, a business profile. A profile row joins a listing row on the same key.
• Business, a 53-column schema, JSON + CSV + a .meta.json sidecar carrying BBB's own result arithmetic.
• 340 offline checks that run with no network and no engine library installed.
• A canary that scrapes 3 real pages daily with no secrets — which is also what keeps the claim above honest.

Three things about BBB that will look like bugs

"Best Match" is not a relevance ordering. BBB's own default returned 15 of 15 BBB Accredited businesses on every page tested and, for find_text=restaurants, not one restaurant. The same query under A-Z returned 0 of 15 accredited and real matches. --sort therefore defaults to a-z — the one place this repo deliberately disagrees with the site — and sort is a column, because it changes which businesses are in the file rather than only their order.

A complete run can be a 1.2% sample. Every query is capped at 15 pages of 15 — 225 rows — however many it matched; one measured search reported 19,016. The sidecar records total_results, pages_available, capped_by_site and reachable_max. Page 16 answers HTTP 500, so runs plan against the site's own number.

An unrated business is not rated zero. rating: "" with ratingScore: 0.0 means not graded — 7 of 105 measured rows — and both columns come back null.

Engine limits, stated rather than discovered

• selenium_scraper.py cannot authenticate a remote CDP endpoint or a proxy, so it refuses a credentialled --cdp-endpoint with exit 2. Wrong engine for --mode profile; perfectly good for the two listing modes.
• puppeteer_scraper.py downloads its own Chromium, which failed to launch during development — --chromium-path points it elsewhere.
• scraper_api_client.py requires --cdp-url on this site: its own exits are datacenter addresses and BBB refuses them.

Not collected, deliberately

BBB names a business's officers as individuals. There is no column for them, and the committed test fixtures carry placeholders: republishing a named person's details is a separate act from the site showing them on its own page.