Repository navigation
v0.1.0 — first release
First release. Two modes over webmotors.com.br's own JSON endpoints, three
browser engines over one shared fetch loop, and the 2Captcha Scraper API
client.
Added
--mode search: a search page of cars (/api/search/car) or motorcycles
(/api/search/bike), one row per listing. Any listing address the site
builds is accepted with--url;--category,--condition,--state,
--makeand--modelbuild one. The five orderings the site's own menu
offers (--sort), and--per-pageup to 100.--mode ad: one row per advert (/api/detail/{car,bike}plus
/api/detail/averageprice), from--urlor--ads-file(a search run's
JSON output, or one address per line). An advert the site answers 404 for
is recorded in the sidecar'sads_goneand does not fail the run.- Row URLs built exactly as the site builds its advert links (660 of 660
identical on the day), so a search's output feeds--mode addirectly. - The currency, read once per run from a listing page's JSON-LD, since no
endpoint states one. - Refusals up front and after page 1 for what the API would otherwise answer
with a WIDER search: an unknown state before anything is sent; a make or
model the site did not apply, from its ownFilterCustomecho. - Sidecar fields
total_results,pages_available,reachable_max,
capped_by_site(the site's ~10,000-result cap),sponsored_skipped,
currencyand, in ad mode,ads_gone. tools/capture.pyandmake_fixtures.py: raw captures, then scrubbed,
trimmed fixtures proven to parse like their originals.
Measured and built in
- CloudFront refuses datacentre addresses; the engines warn when run with no
proxy and no--cdp-endpoint, and retry a refused page three times from a
fresh browser, since a rotating residential gateway gives each one a new
exit (5 of 12 accepted on a-region-brlogin, 9 of 12 on-region-us). - PerimeterX refuses headless Chromium's
HeadlessChromeuser agent (0 of 3)
and an override that drops the client hints (pyppeteer, 0 of 2). All three
engines send a complete identity (page_flow.ua_override). - pyppeteer answers a proxy's authentication over CDP's Fetch domain, since
its ownpage.authenticateis dead on current Chromium, and connects to a
Scraping Browser withoutignoreHTTPSErrors, which hung it. - Sponsored new-car tiles are skipped and do not shift
position.
Not included, on purpose
- No captcha solver. The one challenge the site showed is PerimeterX's Press
& Hold, which this repo does not implement, and no client it drives was
shown it once the identity fixes were in.captcha_solver.pyfrom the
family core is not shipped rather than kept as dead code. - No private seller's name, postal code, phone or description is written.