Skip to content

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors - #317

Closed
LuizFrareWJ wants to merge 1 commit into
adobe:mainfrom
LuizFrareWJ:fix/scrape-webpage-browser-hardening
Closed

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors#317
LuizFrareWJ wants to merge 1 commit into
adobe:mainfrom
LuizFrareWJ:fix/scrape-webpage-browser-hardening

Conversation

@LuizFrareWJ

Copy link
Copy Markdown

Closes #82

Problem

analyze-webpage.js launched a bare chromium.launch() and navigated with page.goto(url) — no launch args, no context identity, no explicit timeouts. Sites that fingerprint headless Chromium, or that reject HTTP/2 from headless clients, fail the scrape before any extraction happens.

Changes

Change Why
Prefer channel: 'chrome', fall back to bundled Chromium Real Chrome carries a genuine TLS fingerprint and a non-headless build string. The fallback is silent, so containers and CI without Chrome keep working.
--disable-http2 Stops ERR_HTTP2_PROTOCOL_ERROR from servers that reject HTTP/2 from headless clients.
domcontentloaded, 60s timeout, 5s settle Pages carrying analytics, chat widgets or ad slots never go network-quiet, so waiting for that only burns the timeout.
Explicit 60s screenshot timeout Full-page screenshots on long pages were unbounded.
Context locale, timezoneId, ignoreHTTPSErrors, User-Agent and sec-ch-ua hints Presents a consistent desktop-Chrome identity.

Two notes on the issue text

The issue was filed in April and two of its bullets no longer apply to main:

  • The hardcoded executablePath is not there. git log -S ms-playwright --all returns nothing, so /ms-playwright/chromium-1208/... never existed in this repository. Only the channel: 'chrome' half of that bullet was actionable.
  • run-bulk-import.js does not exist here, so there was nothing to align the context config to literally. I implemented the properties the bullet names: UA, sec-ch-ua, locale, timezone, ignoreHTTPSErrors.

I also deliberately did not hardcode "Chrome 131". A User-Agent claiming 131 while the browser is a current build is itself a detection signal, and it goes stale on every Chrome release. The UA and the client hints are derived from browser.version(), so they always agree with the browser that actually launched.

While in there: the comment above the navigation claimed it "tried networkidle first", but page.goto(url) with no options waits for load. The comment described behaviour the code never had, so it is gone along with the now-dead fallback branch.

Verification

Against https://www.aem.live/home (macOS, Chrome 152 installed):

"stats": { "total": 28, "converted": 19, "skipped": 9, "failed": 0, "tooLarge": 0 }
9.2s wall clock

Both launch paths exercised:

channel chrome    -> 152.0.7977.65
fallback chromium -> 151.0.7922.34   (invalid channel -> PlaywrightError -> bundled)

Headers as actually sent, captured with a local echo server:

user-agent:          Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36
sec-ch-ua:           "Google Chrome";v="152", "Chromium";v="152", "Not=A?Brand";v="24"
sec-ch-ua-platform:  "macOS"
sec-ch-ua-mobile:    ?0
accept-language:     en-US

npm run validate passes (166 skills, exit 0).

Breaking changes

None. The CLI signature, the returned JSON shape and the emitted files are unchanged.

…and HTTP/2 errors

Prefer the locally installed Google Chrome via `channel: 'chrome'` and fall
back to the Playwright-managed Chromium when it is absent, so the script keeps
working in containers and CI while presenting a real TLS fingerprint elsewhere.

Add --disable-http2 so servers that reject HTTP/2 from headless clients no
longer fail the navigation with ERR_HTTP2_PROTOCOL_ERROR.

Navigate with `domcontentloaded` (60s) plus a 5s settle instead of relying on
the default `load` wait. The previous code commented that it tried `networkidle`
first, but `page.goto(url)` with no options waits for `load`, so the comment
described behaviour the code never had.

Give the full-page screenshot an explicit 60s timeout, and set locale, timezone,
ignoreHTTPSErrors and User-Agent/sec-ch-ua client hints on the browser context.
Those hints are derived from the version of the browser that actually launched
rather than hardcoded, so they agree with each other and do not go stale.

Closes adobe#82

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LuizFrareWJ

Copy link
Copy Markdown
Author

Per CONTRIBUTING.md: this PR was co-authored with an AI tool and should carry the ai-generated label. I can't apply labels as an outside contributor — could a maintainer add it? Disclosing here in the meantime, and the commits carry a Co-Authored-By trailer.

@LuizFrareWJ

Copy link
Copy Markdown
Author

Superseded by #319 — reopening from my personal account (@LuizFrare), which is the correct identity for this contribution. Same commits, same content. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors

2 participants