v1.1.0

Behaviour change: --retries 0 is now a usage error (exit 2). It used
to be accepted and made the page-load loop unreachable, so nothing was
fetched and the engine reported the browser's own blank page as a result:
exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a
script of yours passes 0, pass 1: the flag counts ATTEMPTS, so 1 means
"try once and do not retry".

Behaviour change for existing users: a page BBB serves that cannot be
read now fails the run instead of ending it. A run that used to finish
complete with exit 0 on such a page now finishes partial with exit 6
(rows were gathered) or exit 5 (none were), and its sidecar names the page
in pages_failed with stop_reason: "unreadable_response". A pipeline
that branched on exit 0 was reading an unread page as the end of a listing.

Behaviour change: a URL BBB answers with its own 404 page now stops at
once with stop_reason: "not_found" (exit 5 when nothing was gathered).
It used to wait 30 s, re-fetch, and end as unreadable_response.

Behaviour change: negative values for --pages, --delay,
--retry-delay, --proxy-block-retries and --concurrency (and 0 for
--pages and --concurrency) are now usage errors (exit 2) in every
engine, and so is a negative --retries/--retry-delay in
scraper_api_client.py.

Behaviour change: CSV cells that begin with =, +, -, @, a tab
or a line break are now prefixed with an apostrophe so a spreadsheet does
not run them as formulas. The JSON output is unchanged; the sidecar says
how many cells differ in csv_cells_escaped. Output files are now written
through a temporary file and renamed, keeping the mode open() would give
a new file and the existing mode of a file being replaced.

Correction to v1.0.1. Its README billed the Managed Challenge solve to
the wrong product. It read "Captcha solving (--twocaptcha-key) — for
the Managed Challenge only" with a table marking that challenge
solvable. The challenge is indeed solvable, but not by anything a
--twocaptcha-key buys here: captcha_solver.py builds RecaptchaV2Task,
RecaptchaV2TaskProxyless and RecaptchaV3TaskProxyless and no Turnstile
task at all. What actually clears it is Captcha.setAutoSolve over
--cdp-endpoint, which is the Scraping Browser API — a separately billed
product. A reader following the old README would have set a key, met the
challenge anyway, and had no way to tell why.

Fixed

• The Selenium and pyppeteer engines crashed on a block with no proxy.
  With no --proxy, a blocked page (and, in pyppeteer, a dead exit) reached
  pool.current/pool.advance on None: a traceback, exit 1, and every row
  from earlier pages lost. Both now re-fetch on the same path, as the
  Playwright engine always did, and report exit 3.

• An interrupted write destroyed the previous good output. out.json,
  out.csv and out.meta.json were truncated before being written, so a
  failure halfway left a short or empty file. They are now written to a
  temporary file beside the target, synced and renamed over it.

• The HTTP status never reached the classifier. Playwright and
  pyppeteer now pass the response status, so BBB's 404 is terminal and a
  marker-less 403 is a block. Selenium has no status to pass; BBB's own 404
  title is recognised in the body for it.

• A BBB_URL from .env blocked --text/--location. A URL from the
  environment is now a default that the query flags replace; a TYPED
  --url with query flags is still refused. .env.example ships BBB_URL
  commented out.

• The credential scan judged exclusions on the absolute path, so a
  clone under a directory named venv scanned 0 files and reported clean.
  Exclusions are now relative to the repository, a scan of 0 files fails,
  and .env* copies plus .json, .csv and .html files are read too.

• Dump and capture files were not ignored: X.page3 from
  --dump-html, a raw .html at the repository root, and live/,
  out/-style working directories.

• The profile canary went green without testing anything. With no
  BBB_CDP_ENDPOINT secret it skipped with a notice, and with one it
  passed on exit 3 or 5. It now fails with an error naming the missing
  secret, and every non-zero exit or empty result is red. The listing
  canary needs no secret and stays scheduled; it now also fails explicitly
  on zero rows.

• --solve-captcha help, a docstring and the README's exit-code table
  described refusals and exit 5 inaccurately; corrected. Comments no longer
  point at documents a reader of this repository cannot open.

• --retries 0 fetched nothing and called it a result. The load loop is
  range(1, retries + 1), so 0 never runs it and the engine judged
  about:blank. Three third-party audits of this family ran exactly that and
  reported a working scraper as broken. --retries now goes through
  page_flow.attempts in every entry point; a check reads the source of each
  one and calls the validator, so a new engine declaring the flag with a
  plain int fails the suite.

• A served page that could not be read was taken for the end of the
  listing. unknown (BBB's page, neither the listing payload nor its own
  empty-result marker) left the page outcome ok with no rows, so the loop
  read "added nothing new" as the end of the results. Reproduced with the
  real fetch and finish code and only the browser replaced: page 1 good,
  page 2 with the row container renamed, page 3 good gave complete,
  no_new_products, exit 0, page 3 never requested. The same state on page 1
  was worse: pages 2 and 3 were still fetched and the run reported complete
  over 14 rows instead of 21, the first page lost without a trace. An
  unreadable page is now a failed page in all three engines. A CONFIRMED
  empty result set (totalResults: 0) is unchanged: it is still a finished
  page and still ends a run as complete.

• The captcha budget was advisory. handle_captcha_if_present runs once
  before a page is classified and once after, and only the second call was
  counted, while the first repeats on every block-retry: three solver calls
  against SOLVES_PER_PAGE = 1, reproduced with the solver stubbed and no
  proxy pool. The allowance is now one page_flow.SolveBudget per page,
  spent at the call to the solver itself in all three engines (counting calls
  would have spent it on a call that found no captcha). It does not reset on
  rotation. There is deliberately no run-wide cap on top of it: a run of N
  pages that each need a solve has bought what it needed.

• diff_runs.py compared two different queries. The sort guard caught two
  orderings of one query and nothing caught two queries, so restaurants in
  New York against restaurants in Chicago, both a-z, both complete, was
  accepted and every line of the diff was the change of city. The scope is
  now read from each sidecar's final_url (query text, location, --state,
  category; page, sort, host and www. ignored) and a mismatch is refused,
  with --force as before. A run the user cut at --pages 3 is refused
  against one that read 15 for the same reason; a listing that merely ran
  out of pages is not. diff_runs.py had no behavioural check at all
  (CI ran --help), so its sort and completeness guards are pinned too.

• The Scraper API path (scraper_api_client.py) failed whenever a wait flag
  was given, and never saw the target's status. Measured 2026-09-23 against
  the live /tasks/sync endpoint: waitFor sent as a JSON-encoded string
  (what this client sent) is answered HTTP 422 "params.waitFor must be an
  object" and is still billed ($0.0005); sent as an object it is answered
  HTTP 200. It is now an object. And the response's status is the API's own
  verdict ("success"), while the target site's HTTP code is http_code — the
  client handed status onward, so a target 403/503 never reached the page
  classifier. It now reads http_code, falling back to status only if that
  is an integer. After the fix, one live call (--mode profile --wait-text BBB, the README's profile URL, no --cdp-url) answered HTTP 200, upstream 200, 1 business row.

• Issue templates described Etsy, not BBB. The bug-report hints talked
  about DataDome, t=bv, shop_rating and exit-IP storefronts, the example
  command scraped an Etsy search, and the site-change template pointed at a
  JSON-LD parser this repo does not have. Rewritten from this repo's README:
  the Cloudflare-gated profile mode, the 15-pages-of-15 cap, unrated
  businesses written as null, and the __PRELOADED_STATE__ anchors. The
  bug-report and question templates no longer link a TROUBLESHOOTING.md
  this repo never had.

• --retries help said an empty "hub category" is a correct answer in
  all three engines. BBB has no hub categories; the empty case here is a
  query that matched nothing, and the help now says so. The matching engine
  comments were corrected the same way.

• Comments that described another site's pages as this one's: the
  akamaihd.net hub-page incident is now attributed to the sibling
  tokopedia-scraper where it happened, the pyppeteer engine's single-page
  note no longer talks about a --mode shop this repo does not have, and
  captcha_solver.py no longer points at a "No DataDome solver" section
  that does not exist.

• SECURITY.md said this project has no releases or tags; it has both.

• The access badge's alt text now says what its label already did: the
  listing modes run without an account (profile mode needs a residential
  exit).

• The README now names the product that clears each refusal, and says in
  those words that this repo does not implement TurnstileTaskProxyless
  — nor the init script that captures sitekey, action, cData and
  chlPageData from Cloudflare's single call to turnstile.render(), which
  is the only way to obtain them because they appear nowhere in the served
  HTML. That is a gap in this repo, not in 2Captcha, which solves Turnstile;
  foodpanda-scraper in this family implements the interception.

• Why it is not implemented is now stated with the measurement behind it:
  the listing path is not behind Cloudflare at all, and the profile path —
  which is — already needs a Scraping Browser session to be reachable
  (measured 2026-09-16: the same profile URL returned 403 direct and 200 in
  full through one). On that path the auto-solve clears the challenge before
  a local solver would get a turn.

• Fifteen lines of unreachable code removed from playwright_scraper.py.
  A function's def line had been lost before this repo's first commit,
  leaving its docstring and its try: return page.content() body indented
  into the end of _mask_credentials, where the control flow can never
  arrive. Nothing called it — _content_when_settled below it does the job.
  The same fifteen lines, byte for byte, were in six repos of this family.

• Five dead helpers removed from the engines, two of them calling a
  page_flow API that does not exist in this repo: page_flow.comparable,
  page_flow.next_page_selector and page_flow.next_page_candidates are
  Tokopedia's, and came here with the code — one docstring still described
  Tokopedia's keyword=kopi&search_id= tracking tail. Nothing called any of
  the five. This repo paginates on page_url() and the endpoint's own
  totalPages, which is the arithmetic the site does for us, so a
  selector-walking next-page helper had no role here even had it worked.

• parse_qsl is no longer imported by the two engines that only used it in
  the helper above. Six other unused imports predate this work and are left
  alone deliberately, so this diff stays attributable.

Added

• The signature-binding check no longer swallows a name that is absent.
  It resolved the callee with getattr(owner, attr, None) and skipped
  anything that came back not-callable — so a call into a shared module
  function that does not exist was treated as "nothing to bind" rather
  than as the loudest failure available. That is how three calls into a
  page_flow API this repo does not have sat in two engines under a green
  run of that very check. Absent is now a failure, naming the caller and its
  line. Verified by control: adding a call to page_flow.does_not_exist()
  turns the suite red.

• check_captcha_capability_claims_match_the_code — a guard in both
  directions. It fails the build on a documented claim that a captcha cannot
  be solved, and on a README that credits a Turnstile solve to this repo
  while no Turnstile task type is built. The second is asserted as a PAIRING
  — task type absent ⇒ the README must carry the disclaimer in those words —
  so it cannot go quiet the way a keyword search can. Verified by control:
  flipping the disclaimer turns the suite red.

• A check for a statement the control flow can never reach, and one that
  the existing undefined-name walk cannot see by design: that walk pools
  every binding in a file rather than tracking scopes, so a name used inside
  dead code passes as long as anything else in the module binds it. The new
  check is narrow — a statement after a return/raise/break/continue
  in the SAME block — and across the eighteen repos of this family it found
  six real problems and zero false positives. Verified by control.