v1.1.0

Behaviour change: --retries 0 is now a usage error (exit 2). It used
to be accepted and made the page-load loop unreachable, so nothing was
fetched and the engine reported the browser's own blank page as a result:
exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a
script of yours passes 0, pass 1: the flag counts ATTEMPTS, so 1 means
"try once and do not retry".

Behaviour change for existing users: a page BBB serves that cannot be
read now fails the run instead of ending it. A run that used to finish
complete with exit 0 on such a page now finishes partial with exit 6
(rows were gathered) or exit 5 (none were), and its sidecar names the page
in pages_failed with stop_reason: "unreadable_response". A pipeline
that branched on exit 0 was reading an unread page as the end of a listing.

Behaviour change: a URL BBB answers with its own 404 page now stops at
once with stop_reason: "not_found" (exit 5 when nothing was gathered).
It used to wait 30 s, re-fetch, and end as unreadable_response.

Behaviour change: negative values for --pages, --delay,
--retry-delay, --proxy-block-retries and --concurrency (and 0 for
--pages and --concurrency) are now usage errors (exit 2) in every
engine, and so is a negative --retries/--retry-delay in
scraper_api_client.py.

Behaviour change: CSV cells that begin with =, +, -, @, a tab
or a line break are now prefixed with an apostrophe so a spreadsheet does
not run them as formulas. The JSON output is unchanged; the sidecar says
how many cells differ in csv_cells_escaped. Output files are now written
through a temporary file and renamed, keeping the mode open() would give
a new file and the existing mode of a file being replaced.

Correction to v1.0.1. Its README billed the Managed Challenge solve to
the wrong product. It read "Captcha solving (--twocaptcha-key) — for
the Managed Challenge only" with a table marking that challenge
solvable. The challenge is indeed solvable, but not by anything a
--twocaptcha-key buys here: captcha_solver.py builds RecaptchaV2Task,
RecaptchaV2TaskProxyless and RecaptchaV3TaskProxyless and no Turnstile
task at all. What actually clears it is Captcha.setAutoSolve over
--cdp-endpoint, which is the Scraping Browser API — a separately billed
product. A reader following the old README would have set a key, met the
challenge anyway, and had no way to tell why.

Fixed

• The Selenium and pyppeteer engines crashed on a block with no proxy.
  With no --proxy, a blocked page (and, in pyppeteer, a dead exit) reached
  pool.current/pool.advance on None: a traceback, exit 1, and every row
  from earlier pages lost. Both now re-fetch on the same path, as the
  Playwright engine always did, and report exit 3.

• An interrupted write destroyed the previous good output. out.json,
  out.csv and out.meta.json were truncated before being written, so a
  failure halfway left a short or empty file. They are now written to a
  temporary file beside the target, synced and renamed over it.

• The HTTP status never reached the classifier. Playwright and
  pyppeteer now pass the response status, so BBB's 404 is terminal and a
  marker-less 403 is a block. Selenium has no status to pass; BBB's own 404
  title is recognised in the body for it.

• A BBB_URL from .env blocked --text/--location. A URL from the
  environment is now a default that the query flags replace; a TYPED
  --url with query flags is still refused. .env.example ships BBB_URL
  commented out.

• The credential scan judged exclusions on the absolute path, so a
  clone under a directory named venv scanned 0 files and reported clean.
  Exclusions are now relative to the repository, a scan of 0 files fails,
  and .env* copies plus .json, .csv and .html files are read too.

• Dump and capture files were not ignored: X.page3 from
  --dump-html, a raw .html at the repository root, and live/,
  out/-style working directories.

• The profile canary went green without testing anything. With no
  BBB_CDP_ENDPOINT secret it skipped with a notice, and with one it
  passed on exit 3 or 5. It now fails with an error naming the missing
  secret, and every non-zero exit or empty result is red. The listing
  canary needs no secret and stays scheduled; it now also fails explicitly
  on zero rows.

• --solve-captcha help, a docstring and the README's exit-code table
  described refusals and exit 5 inaccurately; corrected. Comments no longer
  point at documents a reader of this repository cannot open.

• --retries 0 fetched nothing and called it a result. The load loop is
  range(1, retries + 1), so 0 never runs it and the engine judged
  about:blank. Three third-party audits of this family ran exactly that and
  reported a working scraper as broken. --retries now goes through
  page_flow.attempts in every entry point; a check reads the source of each
  one and calls the validator, so a new engine declaring the flag with a
  plain int fails the suite.

• A served page that could not be read was taken for the end of the
  listing. unknown (BBB's page, neither the listing payload nor its own
  empty-result marker) left the page outcome ok with no rows, so the loop
  read "added nothing new" as the end of the results. Reproduced with the
  real fetch and finish code and only the browser replaced: page 1 good,
  page 2 with the row container renamed, page 3 good gave complete,
  no_new_products, exit 0, page 3 never requested. The same state on page 1
  was worse: pages 2 and 3 were still fetched and the run reported complete
  over 14 rows instead of 21, the first page lost without a trace. An
  unreadable page is now a failed page in all three engines. A CONFIRMED
  empty result set (totalResults: 0) is unchanged: it is still a finished
  page and still ends a run as complete.

• The captcha budget was advisory. handle_captcha_if_present runs once
  before a page is classified and once after, and only the second call was
  counted, while the first repeats on every block-retry: three solver calls
  against SOLVES_PER_PAGE = 1, reproduced with the solver stubbed and no
  proxy pool. The allowance is now one page_flow.SolveBudget per page,
  spent at the call to the solver itself in all three engines (counting calls
  would have spent it on a call that found no captcha). It does not reset on
  rotation. There is deliberately no run-wide cap on top of it: a run of N
  pages that each need a solve has bought what it needed.

• diff_runs.py compared two different queries. The sort guard caught two
  orderings of one query and nothing caught two queries, so restaurants in
  New York against restaurants in Chicago, both a-z, both complete, was
  accepted and every line of the diff was the change of city. The scope is
  now read from each sidecar's final_url (query text, location, --state,
  category; page, sort, host and www. ignored) and a mismatch is refused,
  with --force as before. A run the user cut at --pages 3 is refused
  against one that read 15 for the same reason; a listing that merely ran
  out of pages is not. diff_runs.py had no behavioural check at all
  (CI ran --help), so its sort and completeness guards are pinned too.

• The Scraper API path (scraper_api_client.py) failed whenever a wait flag
  was given, and never saw the target's status. Measured 2026-09-23 against
  the live /tasks/sync endpoint: waitFor sent as a JSON-encoded string
  (what this client sent) is answered HTTP 422 "params.waitFor must be an
  object" and is still billed ($0.0005); sent as an object it is answered
  HTTP 200. It is now an object. And the response's status is the API's own
  verdict ("success"), while the target site's HTTP code is http_code — the
  client handed status onward, so a target 403/503 never reached the page
  classifier. It now reads http_code, falling back to status only if that
  is an integer. After the fix, one live call (--mode profile --wait-text BBB, the README's profile URL, no --cdp-url) answered HTTP 200, upstream 200, 1 business row.

• Issue templates described Etsy, not BBB. The bug-report hints talked
  about DataDome, t=bv, shop_rating and exit-IP storefronts, the example
  command scraped an Etsy search, and the site-change template pointed at a
  JSON-LD parser this repo does not have. Rewritten from this repo's README:
  the Cloudflare-gated profile mode, the 15-pages-of-15 cap, unrated
  businesses written as null, and the __PRELOADED_STATE__ anchors. The
  bug-report and question templates no longer link a TROUBLESHOOTING.md
  this repo never had.

• --retries help said an empty "hub category" is a correct answer in
  all three engines. BBB has no hub categories; the empty case here is a
  query that matched nothing, and the help now says so. The matching engine
  comments were corrected the same way.

• Comments that described another site's pages as this one's: the
  akamaihd.net hub-page incident is now attributed to the sibling
  tokopedia-scraper where it happened, the pyppeteer engine's single-page
  note no longer talks about a --mode shop this repo does not have, and
  captcha_solver.py no longer points at a "No DataDome solver" section
  that does not exist.

• SECURITY.md said this project has no releases or tags; it has both.

• The access badge's alt text now says what its label already did: the
  listing modes run without an account (profile mode needs a residential
  exit).

• The README now names the product that clears each refusal, and says in
  those words that this repo does not implement TurnstileTaskProxyless
  — nor the init script that captures sitekey, action, cData and
  chlPageData from Cloudfla...

v1.0.1 — a marker that fired on good pages

A challenge marker was wrong, and it was one that could have cost money. cf-turnstile fired on five of five pages BBB actually served. Only the signal ordering in detect_page_state — the listing payload is checked before any marker — kept a good page from being reported as a challenge, and challenge is the state that spends.

What was measured

Counted across every capture taken 2026-09-16:

	served (6)	challenge (2)	hard block
cf_chl_opt	0	7, 7	0
__cf_chl	0	3, 3	0
cf-chl-	0	1, 0	0
challenges.cloudflare.com	0	1, 0	0
cf-turnstile	1 on 5!	1, 0	0
/turnstile/v0/api.js	0	0, 0	0

The cause is not BBB. 2Captcha's Scraping Browser auto-solve extension injects chrome-extension://kjmkgkdkpedkejedfhmfcenooemhbpbo/…/turnstile/hunter.js with a cf-turnstile-response input into every page it loads, and every capture in this repo was taken that way.

Fixed

• cf-turnstile and /turnstile/v0/api.js removed from BOT_CHALLENGE_MARKERS; challenges.cloudflare.com added.
• The marker guard in the suite had been passing for the wrong reason — it ran only against BBB's 404, fetched with plain curl, which carries no extension injection. It now also runs against a listing fetched through the Scraping Browser, and asserts in both directions: no marker on a served page, every marker firing on a real challenge, and each excluded string really present on a served page.

352 offline checks, up from 340.

Which captcha this site actually uses

BBB sits behind Cloudflare and refuses in two shapes — both HTTP 403, both titled … | Better Business Bureau®:

	what it is	solvable
Just a moment...	a Cloudflare Managed Challenge (Turnstile)	yes — a real test
You have been blocked	a flat refusal: no widget, no sitekey	no — a solve buys nothing

Separately, BBB has its own reCAPTCHA Enterprise on every page (render=<sitekey>, so v3 rather than a v2 checkbox), guarding its review and complaint forms. This scraper never touches those forms — so no captcha was ever solved, and nothing was ever spent on one.

v1.0.0 — BBB scraper, rebuilt

First release of the rebuilt scraper. It replaces four standalone scripts that predated the template — no shared row model, no exit-code contract, no sidecar, no tests.

Read this first

--mode search and --mode category need no account, no key and no proxy.

BBB renders its listings out of its own JSON endpoint, and that endpoint is not behind Cloudflare. Measured 2026-09-16 from a datacenter VPS — an address that gets HTTP 403 on every HTML page on the site — the endpoint answered HTTP 200 with 15 complete business records. A two-page run from that machine, with no credentials configured at all, returned 30 rows and status: complete.

--mode profile is the one that needs a residential exit, and it is what the paid products are for here: a profile page is gated, and there is no endpoint behind it.

What's in it

• Three engines — Playwright (primary, the only one with --concurrency), Selenium, pyppeteer — plus a browserless client for 2Captcha's Scraper API. All produce identical rows and exit codes.
• Three modes: a keyword search, a category listing, a business profile. A profile row joins a listing row on the same key.
• Business, a 53-column schema, JSON + CSV + a .meta.json sidecar carrying BBB's own result arithmetic.
• 340 offline checks that run with no network and no engine library installed.
• A canary that scrapes 3 real pages daily with no secrets — which is also what keeps the claim above honest.

Three things about BBB that will look like bugs

"Best Match" is not a relevance ordering. BBB's own default returned 15 of 15 BBB Accredited businesses on every page tested and, for find_text=restaurants, not one restaurant. The same query under A-Z returned 0 of 15 accredited and real matches. --sort therefore defaults to a-z — the one place this repo deliberately disagrees with the site — and sort is a column, because it changes which businesses are in the file rather than only their order.

A complete run can be a 1.2% sample. Every query is capped at 15 pages of 15 — 225 rows — however many it matched; one measured search reported 19,016. The sidecar records total_results, pages_available, capped_by_site and reachable_max. Page 16 answers HTTP 500, so runs plan against the site's own number.

An unrated business is not rated zero. rating: "" with ratingScore: 0.0 means not graded — 7 of 105 measured rows — and both columns come back null.

Engine limits, stated rather than discovered

• selenium_scraper.py cannot authenticate a remote CDP endpoint or a proxy, so it refuses a credentialled --cdp-endpoint with exit 2. Wrong engine for --mode profile; perfectly good for the two listing modes.
• puppeteer_scraper.py downloads its own Chromium, which failed to launch during development — --chromium-path points it elsewhere.
• scraper_api_client.py requires --cdp-url on this site: its own exits are datacenter addresses and BBB refuses them.

Not collected, deliberately

BBB names a business's officers as individuals. There is no column for them, and the committed test fixtures carry placeholders: republishing a named person's details is a separate act from the site showing them on its own page.