Repository navigation
v1.0.1
Fixes from an external audit of v1.0.0, and from live runs on 2026-10-08,
-09 and -10. Each finding was reproduced before it was fixed, and each now
has a regression test.
Behaviour changes for existing users.
- An unexpected exception now exits 1 (crash, traceback printed
with credentials masked). It used to exit 5, the same as a dead exit, so a
parser bug read as a transport fault. Exit 5 is now only a transport fault.diff_runs.pynow refuses two outputs when either has no readable
.meta.json, or when they were written by different row schemas. Pass
--forceto compare anyway.--mode sellernow exits 6 (partial) when it wrote fewer listings
than the profile's own counter, instead of 0.
Fixed
- The installed wheel allowed every URL. The robots snapshot was a loose
.txtbeside the modules and was not shipped; an installed copy read zero
rules. It is now the modulerobots_snapshot.py, and an empty rule set
raises instead of allowing everything. pip install avito-scrapergave a command that crashed with
ModuleNotFoundError: playwright. The command now goes through a launcher
that says to installavito-scraper[playwright](exit 2).- No attempt metadata on a failed start or a crash.
<out>.latest_attempt.meta.jsonis now written on every ending, with
exit_code,error_classand a maskederror_message. - A failed captcha solve is now traceable. Every solve in
transport_facts.solvesrecords the 2Captchatask_idand its own error
code (ERROR_CAPTCHA_UNSOLVABLE,TIMEOUT, …), and the solver waits 240 s
instead of 180 s: on 2026-10-08 2Captcha took 217–220 s to return
ERROR_CAPTCHA_UNSOLVABLEfor this captcha, so the old wait abandoned
every task before its verdict and logged only "no solution". - A third firewall variant — «подождите немного и обновите страницу», no
captcha form, no proof-of-work, served right after an accepted solve
(measured 2026-10-08) — is now a refusal of the exit and rotates, instead
of being waited out as a challenge and reported blocked. - A solve is
verifiedonly when the page actually came back. The first
release marked it verified as soon as the document stopped being the wall,
which included that third variant. A solve that is followed by something
else recordsafter_submit. A proof-of-work after a solve is waited out. - The wall's other two captchas. The page can show Avito's own picture
captcha or hCaptcha instead of GeeTest; the scraper now recognises which
one is showing and records it (solves[].variant). The picture captcha is
solved withImageToTextTaskand answered through the page's own field —
implemented from the wall's script, not yet met live (every measured
request was assigned GeeTest). hCaptcha is solved withHCaptchaTask
(sitekey from the wall, the browser's own exit,enterprisePayload.rqdata
only if the wall ever carries one) and its token submitted through
#h-captcha-response— also not yet met live. One liveHCaptchaTaskon
avito's sitekey (2026-10-09) was accepted by the API and returned
ERROR_CAPTCHA_UNSOLVABLE after 51 s: the request shape is right, a solved
token for this wall is not yet measured. --mode sellernow returns all of a seller's listings, not the 15
the profile renders: it opens the seller's feed, presses «Показать все»
and scrolls until the profile's own counter or--max-products
(measured: 353 of 353). The feed's scroll makes the page fetch
/web/1/profile/items, which robots.txt disallows — README says so; this
is a deliberate choice.--max-products 15keeps to the profile page.- A recommendation carousel was read as search results. On 2026-10-10
?q=yeezypage 1 carried a «Подобрали для вас» block INSIDE the results
grid, with 31 cards of the same shape; page 1 came back with 82 rows, 32
of them unrelated. Cards insideitemsCarouselare no longer results. - A seller run with fewer listings than the profile states is now
partial(exit 6). A shop stating 7331 active listings was written as a
completerun of 12, because the profile's tabbed counter («Активные |
N») was not read and the feed never started. Both are fixed; a shop
profile without asellerIdnow takes it from/brands/<slug>/items
(measured: 100 of 100 with--max-products 100). - The seller's name falls back to the profile heading when the page title
does not carry it (one fetch of two, 2026-10-10). - Rotation messages name Chromium's error (
ERR_CONNECTION_RESET) instead of
a fragment of the URL. schema_versionnow fingerprints the row class of the run's mode
(names, order and types), notProduct's field names only.- Leftovers from the repositories this one was scaffolded from, in
requirements.txt,.gitignore,.dockerignoreand the Claude review
prompt, are gone; the vocabulary check now covers every tracked file.
Comments no longer point at an operator file that is not published.
Changed
- CI builds the wheel, installs it outside the checkout and uses it
(ci_checks.py --wheel-check): robots verdicts, and the console command
with and without the playwright extra. - A skipped canary (no secrets — the case for this public repository) now
raises a visible warning; README explains what the green badge does and
does not prove.
Not changed, and why
- The canary still goes green when it skips. A check that is red every
day for want of secrets teaches everyone to ignore checks; the skip is now
loud instead. - Pinning Actions by SHA, lock files, a
src/package layout and dropping
pyppeteer are larger decisions, left for a minor release.