Skip to content

Heavily-protected sites (CARFAX / Cloudflare-class) blocked by client-side bot detection #189

Description

@us

Summary

Reported by @itsSteven_Phan: scraping a CARFAX vehicle-history report returns
"Looks like your internet browser isn't supported" instead of the content.

Repro: crw scrape "https://vhr.carfax.ca/?id=..."

What we ruled out (it is NOT the User-Agent)

The v0.18.0→v0.18.3 work fixed UA presentation across every path:

  • #176 — HTTP-only fetcher → Chrome 150
  • v0.18.2 — CDP path (Network.setUserAgentOverride)
  • v0.18.3 — lightpanda (strips the Mozilla/5.0 prefix it rejects)

But CARFAX still blocks us. Evidence it's not UA-based:

  • HTTP test: every UA (full Chrome, stripped, lightpanda, even curl) gets
    the same 200 + identical 22 KB shell — no HTTP-level UA gating.
  • Prod cloud runs engineVersion 0.18.3 (real Chromium + Chrome 150 UA)
    and still shows "unsupported".

Actual cause

Client-side bot / headless detection (think navigator.webdriver, headless
flags, TLS/JA3 fingerprint, etc.) — plus the report itself loads asynchronously
("collecting data from thousands of sources…"). This is a different, harder
problem class than a UA string, and a real headless Chrome with a correct UA is
blocked all the same.

Direction (not done yet)

A proper stealth / anti-detect tier: harden the existing STEALTH_JS
(webdriver/plugins masking), pair with residential proxy egress, and handle
async-rendered reports. No promises on fully-protected sites like CARFAX, but
this is the right track. Tracking scope + progress here.

Thanks @itsSteven_Phan for the detailed repro 🙏

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions