Summary
Reported by @itsSteven_Phan: scraping a CARFAX vehicle-history report returns
"Looks like your internet browser isn't supported" instead of the content.
Repro: crw scrape "https://vhr.carfax.ca/?id=..."
What we ruled out (it is NOT the User-Agent)
The v0.18.0→v0.18.3 work fixed UA presentation across every path:
#176 — HTTP-only fetcher → Chrome 150
v0.18.2 — CDP path (Network.setUserAgentOverride)
v0.18.3 — lightpanda (strips the Mozilla/5.0 prefix it rejects)
But CARFAX still blocks us. Evidence it's not UA-based:
- HTTP test: every UA (full Chrome, stripped, lightpanda, even
curl) gets
the same 200 + identical 22 KB shell — no HTTP-level UA gating.
- Prod cloud runs
engineVersion 0.18.3 (real Chromium + Chrome 150 UA)
and still shows "unsupported".
Actual cause
Client-side bot / headless detection (think navigator.webdriver, headless
flags, TLS/JA3 fingerprint, etc.) — plus the report itself loads asynchronously
("collecting data from thousands of sources…"). This is a different, harder
problem class than a UA string, and a real headless Chrome with a correct UA is
blocked all the same.
Direction (not done yet)
A proper stealth / anti-detect tier: harden the existing STEALTH_JS
(webdriver/plugins masking), pair with residential proxy egress, and handle
async-rendered reports. No promises on fully-protected sites like CARFAX, but
this is the right track. Tracking scope + progress here.
Thanks @itsSteven_Phan for the detailed repro 🙏
Summary
Reported by @itsSteven_Phan: scraping a CARFAX vehicle-history report returns
"Looks like your internet browser isn't supported" instead of the content.
Repro:
crw scrape "https://vhr.carfax.ca/?id=..."What we ruled out (it is NOT the User-Agent)
The v0.18.0→v0.18.3 work fixed UA presentation across every path:
#176— HTTP-only fetcher → Chrome 150v0.18.2— CDP path (Network.setUserAgentOverride)v0.18.3— lightpanda (strips theMozilla/5.0prefix it rejects)But CARFAX still blocks us. Evidence it's not UA-based:
curl) getsthe same
200+ identical 22 KB shell — no HTTP-level UA gating.engineVersion 0.18.3(real Chromium + Chrome 150 UA)and still shows "unsupported".
Actual cause
Client-side bot / headless detection (think
navigator.webdriver, headlessflags, TLS/JA3 fingerprint, etc.) — plus the report itself loads asynchronously
("collecting data from thousands of sources…"). This is a different, harder
problem class than a UA string, and a real headless Chrome with a correct UA is
blocked all the same.
Direction (not done yet)
A proper stealth / anti-detect tier: harden the existing
STEALTH_JS(webdriver/plugins masking), pair with residential proxy egress, and handle
async-rendered reports. No promises on fully-protected sites like CARFAX, but
this is the right track. Tracking scope + progress here.
Thanks @itsSteven_Phan for the detailed repro 🙏