Skip to content

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors - #319

Open
LuizFrare wants to merge 1 commit into
adobe:mainfrom
LuizFrare:fix/scrape-webpage-browser-hardening
Open

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors#319
LuizFrare wants to merge 1 commit into
adobe:mainfrom
LuizFrare:fix/scrape-webpage-browser-hardening

Conversation

@LuizFrare

@LuizFrare LuizFrare commented Aug 31, 2026

Copy link
Copy Markdown

Closes #82

Problem

analyze-webpage.js launched a bare chromium.launch() and navigated with page.goto(url) — no launch args, no context identity, no explicit timeouts. Sites that fingerprint headless Chromium, or that reject HTTP/2 from headless clients, fail the scrape before any extraction happens.

Changes

Change Why
Prefer channel: 'chrome', fall back to bundled Chromium Real Chrome carries a genuine TLS fingerprint and a non-headless build string. The fallback is silent, so containers and CI without Chrome keep working.
--disable-http2 Stops ERR_HTTP2_PROTOCOL_ERROR from servers that reject HTTP/2 from headless clients.
domcontentloaded with a 60s timeout Pages carrying analytics, chat widgets or ad slots never go network-quiet, so waiting for that only burns the timeout.
Settle by awaiting load capped at 5s, not a fixed 5s sleep Same protection for deferred content without charging every fast page five seconds. See the measurement below.
Explicit 60s screenshot timeout Full-page screenshots on long pages were unbounded.
Context locale, timezoneId, ignoreHTTPSErrors, User-Agent and sec-ch-ua hints Presents a consistent desktop-Chrome identity.
SKILL.md Prerequisites and Troubleshooting updated The launch behaviour changed, so the skill's own docs had to say so.

Two notes on the issue text

The issue was filed in April and two of its bullets no longer apply to main:

  • The hardcoded executablePath is not there. git log -S ms-playwright --all returns nothing, so /ms-playwright/chromium-1208/... never existed in this repository. Only the channel: 'chrome' half of that bullet was actionable.
  • run-bulk-import.js does not exist here, so there was nothing to align the context config to literally. I implemented the properties the bullet names: UA, sec-ch-ua, locale, timezone, ignoreHTTPSErrors.

Two deliberate departures from the issue's literal wording:

  • No hardcoded "Chrome 131" UA. A User-Agent claiming 131 while the browser is a current build is itself a detection signal, and it goes stale on every Chrome release. The UA and the client hints are derived from browser.version(), so they always agree with the browser that actually launched.
  • The 5s settle is a cap, not a sleep. page.waitForLoadState('load', { timeout: 5000 }) gives deferred content the same window while letting fast pages continue immediately.

While in there: the comment above the navigation claimed it "tried networkidle first", but page.goto(url) with no options waits for load. The comment described behaviour the code never had, so it is gone along with the now-dead fallback branch.

Verification

Against https://www.aem.live/home (macOS, Chrome 152 installed), three warm runs each:

run 1 run 2 run 3
fixed 5s sleep 7.11s 7.15s 7.15s
load capped at 5s 2.49s 2.18s 2.17s

Output is unchanged — cleaned.html is byte-for-byte identical between the two, with the same image stats and the same 18 metadata keys:

{"total":28,"converted":19,"skipped":9,"failed":0,"tooLarge":0,"limitReached":0}

Both launch paths exercised:

channel chrome    -> 152.0.7977.65
fallback chromium -> 151.0.7922.34   (invalid channel -> PlaywrightError -> bundled)

Headers as actually sent, captured with a local echo server:

user-agent:          Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36
sec-ch-ua:           "Google Chrome";v="152", "Chromium";v="152", "Not=A?Brand";v="24"
sec-ch-ua-platform:  "macOS"
sec-ch-ua-mobile:    ?0
accept-language:     en-US

npm run validate passes (166 skills, exit 0).

Breaking changes

None. The CLI signature, the returned JSON shape and the emitted files are unchanged.

@LuizFrare

Copy link
Copy Markdown
Author

Per CONTRIBUTING.md: this PR was co-authored with an AI tool and should carry the ai-generated label. I can't apply labels as an outside contributor — could a maintainer add it? Disclosing here in the meantime, and the commits carry a Co-Authored-By trailer.

…and HTTP/2 errors

Prefer the locally installed Google Chrome via `channel: 'chrome'` and fall
back to the Playwright-managed Chromium when it is absent, so the script keeps
working in containers and CI while presenting a real TLS fingerprint elsewhere.

Add --disable-http2 so servers that reject HTTP/2 from headless clients no
longer fail the navigation with ERR_HTTP2_PROTOCOL_ERROR.

Navigate with `domcontentloaded` (60s) instead of relying on the default `load`
wait. The previous code commented that it tried `networkidle` first, but
`page.goto(url)` with no options waits for `load`, so the comment described
behaviour the code never had.

Settle after navigation by waiting for `load` with a 5s cap rather than sleeping
for a fixed 5s: a page that gets there quickly continues immediately, and one
that never does is capped instead of waited out.

Give the full-page screenshot an explicit 60s timeout, and set locale, timezone,
ignoreHTTPSErrors and User-Agent/sec-ch-ua client hints on the browser context.
Those hints are derived from the version of the browser that actually launched
rather than hardcoded, so they agree with each other and do not go stale.

Document the Chrome-first behaviour in the skill's Prerequisites and
Troubleshooting sections, since it changes what the skill needs installed.

Closes adobe#82

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LuizFrare
LuizFrare force-pushed the fix/scrape-webpage-browser-hardening branch from 7a8bf81 to c58461b Compare August 31, 2026 21:05
@LuizFrare

Copy link
Copy Markdown
Author

The workflow runs on this PR are sitting in action_required, so validate and conventional-commits have not reported yet. As an outside contributor I cannot trigger them myself. Could a maintainer approve the runs?

The CLA check is signed and green. I ran npm run validate locally against the full repo and it passed (166 skills, exit 0), and the PR body has the before/after timings plus a byte-identical output check, but CI confirming it is worth more than me claiming it.

Glad to adjust anything here, including the two places where I departed from the literal wording of #82 (no hardcoded Chrome 131 UA, and the 5s settle implemented as a cap rather than a sleep). Both are explained in the description, and I am happy to go the other way if you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors

1 participant