fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors - #317
Closed
LuizFrareWJ wants to merge 1 commit into
Closed
Conversation
…and HTTP/2 errors Prefer the locally installed Google Chrome via `channel: 'chrome'` and fall back to the Playwright-managed Chromium when it is absent, so the script keeps working in containers and CI while presenting a real TLS fingerprint elsewhere. Add --disable-http2 so servers that reject HTTP/2 from headless clients no longer fail the navigation with ERR_HTTP2_PROTOCOL_ERROR. Navigate with `domcontentloaded` (60s) plus a 5s settle instead of relying on the default `load` wait. The previous code commented that it tried `networkidle` first, but `page.goto(url)` with no options waits for `load`, so the comment described behaviour the code never had. Give the full-page screenshot an explicit 60s timeout, and set locale, timezone, ignoreHTTPSErrors and User-Agent/sec-ch-ua client hints on the browser context. Those hints are derived from the version of the browser that actually launched rather than hardcoded, so they agree with each other and do not go stale. Closes adobe#82 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
|
Per CONTRIBUTING.md: this PR was co-authored with an AI tool and should carry the |
Author
|
Superseded by #319 — reopening from my personal account (@LuizFrare), which is the correct identity for this contribution. Same commits, same content. Sorry for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #82
Problem
analyze-webpage.jslaunched a barechromium.launch()and navigated withpage.goto(url)— no launch args, no context identity, no explicit timeouts. Sites that fingerprint headless Chromium, or that reject HTTP/2 from headless clients, fail the scrape before any extraction happens.Changes
channel: 'chrome', fall back to bundled Chromium--disable-http2ERR_HTTP2_PROTOCOL_ERRORfrom servers that reject HTTP/2 from headless clients.domcontentloaded, 60s timeout, 5s settlelocale,timezoneId,ignoreHTTPSErrors, User-Agent andsec-ch-uahintsTwo notes on the issue text
The issue was filed in April and two of its bullets no longer apply to
main:executablePathis not there.git log -S ms-playwright --allreturns nothing, so/ms-playwright/chromium-1208/...never existed in this repository. Only thechannel: 'chrome'half of that bullet was actionable.run-bulk-import.jsdoes not exist here, so there was nothing to align the context config to literally. I implemented the properties the bullet names: UA,sec-ch-ua, locale, timezone,ignoreHTTPSErrors.I also deliberately did not hardcode "Chrome 131". A User-Agent claiming 131 while the browser is a current build is itself a detection signal, and it goes stale on every Chrome release. The UA and the client hints are derived from
browser.version(), so they always agree with the browser that actually launched.While in there: the comment above the navigation claimed it "tried networkidle first", but
page.goto(url)with no options waits forload. The comment described behaviour the code never had, so it is gone along with the now-dead fallback branch.Verification
Against
https://www.aem.live/home(macOS, Chrome 152 installed):Both launch paths exercised:
Headers as actually sent, captured with a local echo server:
npm run validatepasses (166 skills, exit 0).Breaking changes
None. The CLI signature, the returned JSON shape and the emitted files are unchanged.