What happens
isChallengeResponse in src/fetch/classify.ts:3 builds its evidence string from all response headers plus the body, then matches challengeMarkers (cloudflare|cf-chl|datadome|perimeterx|px-captcha|akamai|captcha|just a moment|verify you are human).
Hacker News serves a perfectly good 200 whose content-security-policy header mentions https://cdnjs.cloudflare.com/ and https://www.google.com/recaptcha/. Both markers hit, so a healthy page is classified as a challenge.
plain 200 challenge=true jsShell=false
content-security-policy: default-src 'self'; script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ … https://cdnjs.cloudflare.com/; …
Consequence: the plain tier already had the full 34 KB of usable HTML, but the impit ladder runs anyway (chrome, then firefox), both fetches return the same 200 that is again "a challenge", and the command ends in:
$ webcmd web fetch --url https://news.ycombinator.com
CliError: The site blocked non-browser fetches.
hint: Use webcmd web fetch-browser for this URL.
The page was never blocked.
Why it matters
Any site whose CSP allow-list names a CDN or reCAPTCHA — a very large share of the web — pays two wasted upstream fetches and then fails with a misleading FETCH_BLOCKED telling the agent to escalate to a browser it does not need.
Suggested fix
Narrow the evidence. Header matching should look at the headers that actually signal a challenge (server, cf-mitigated, cf-chl-*, x-datadome, set-cookie names like __cf_bm/datadome) rather than every header value, and CSP/report-to/link should be excluded outright — they are allow-lists of third parties, not evidence about this response. Body matching can stay as-is.
Found while fixing #249; the two are independent (that one is about the retry ladder blowing the --timeout budget, this one is about the ladder running at all).
What happens
isChallengeResponseinsrc/fetch/classify.ts:3builds its evidence string from all response headers plus the body, then matcheschallengeMarkers(cloudflare|cf-chl|datadome|perimeterx|px-captcha|akamai|captcha|just a moment|verify you are human).Hacker News serves a perfectly good
200whosecontent-security-policyheader mentionshttps://cdnjs.cloudflare.com/andhttps://www.google.com/recaptcha/. Both markers hit, so a healthy page is classified as a challenge.Consequence: the plain tier already had the full 34 KB of usable HTML, but the impit ladder runs anyway (chrome, then firefox), both fetches return the same 200 that is again "a challenge", and the command ends in:
The page was never blocked.
Why it matters
Any site whose CSP allow-list names a CDN or reCAPTCHA — a very large share of the web — pays two wasted upstream fetches and then fails with a misleading
FETCH_BLOCKEDtelling the agent to escalate to a browser it does not need.Suggested fix
Narrow the evidence. Header matching should look at the headers that actually signal a challenge (
server,cf-mitigated,cf-chl-*,x-datadome,set-cookienames like__cf_bm/datadome) rather than every header value, and CSP/report-to/linkshould be excluded outright — they are allow-lists of third parties, not evidence about this response. Body matching can stay as-is.Found while fixing #249; the two are independent (that one is about the retry ladder blowing the
--timeoutbudget, this one is about the ladder running at all).