fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors - #319
fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors#319LuizFrare wants to merge 1 commit into
Conversation
|
Per CONTRIBUTING.md: this PR was co-authored with an AI tool and should carry the |
…and HTTP/2 errors Prefer the locally installed Google Chrome via `channel: 'chrome'` and fall back to the Playwright-managed Chromium when it is absent, so the script keeps working in containers and CI while presenting a real TLS fingerprint elsewhere. Add --disable-http2 so servers that reject HTTP/2 from headless clients no longer fail the navigation with ERR_HTTP2_PROTOCOL_ERROR. Navigate with `domcontentloaded` (60s) instead of relying on the default `load` wait. The previous code commented that it tried `networkidle` first, but `page.goto(url)` with no options waits for `load`, so the comment described behaviour the code never had. Settle after navigation by waiting for `load` with a 5s cap rather than sleeping for a fixed 5s: a page that gets there quickly continues immediately, and one that never does is capped instead of waited out. Give the full-page screenshot an explicit 60s timeout, and set locale, timezone, ignoreHTTPSErrors and User-Agent/sec-ch-ua client hints on the browser context. Those hints are derived from the version of the browser that actually launched rather than hardcoded, so they agree with each other and do not go stale. Document the Chrome-first behaviour in the skill's Prerequisites and Troubleshooting sections, since it changes what the skill needs installed. Closes adobe#82 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7a8bf81 to
c58461b
Compare
|
The workflow runs on this PR are sitting in The CLA check is signed and green. I ran Glad to adjust anything here, including the two places where I departed from the literal wording of #82 (no hardcoded Chrome 131 UA, and the 5s settle implemented as a cap rather than a sleep). Both are explained in the description, and I am happy to go the other way if you prefer. |
Closes #82
Problem
analyze-webpage.jslaunched a barechromium.launch()and navigated withpage.goto(url)— no launch args, no context identity, no explicit timeouts. Sites that fingerprint headless Chromium, or that reject HTTP/2 from headless clients, fail the scrape before any extraction happens.Changes
channel: 'chrome', fall back to bundled Chromium--disable-http2ERR_HTTP2_PROTOCOL_ERRORfrom servers that reject HTTP/2 from headless clients.domcontentloadedwith a 60s timeoutloadcapped at 5s, not a fixed 5s sleeplocale,timezoneId,ignoreHTTPSErrors, User-Agent andsec-ch-uahintsSKILL.mdPrerequisites and Troubleshooting updatedTwo notes on the issue text
The issue was filed in April and two of its bullets no longer apply to
main:executablePathis not there.git log -S ms-playwright --allreturns nothing, so/ms-playwright/chromium-1208/...never existed in this repository. Only thechannel: 'chrome'half of that bullet was actionable.run-bulk-import.jsdoes not exist here, so there was nothing to align the context config to literally. I implemented the properties the bullet names: UA,sec-ch-ua, locale, timezone,ignoreHTTPSErrors.Two deliberate departures from the issue's literal wording:
browser.version(), so they always agree with the browser that actually launched.page.waitForLoadState('load', { timeout: 5000 })gives deferred content the same window while letting fast pages continue immediately.While in there: the comment above the navigation claimed it "tried networkidle first", but
page.goto(url)with no options waits forload. The comment described behaviour the code never had, so it is gone along with the now-dead fallback branch.Verification
Against
https://www.aem.live/home(macOS, Chrome 152 installed), three warm runs each:loadcapped at 5sOutput is unchanged —
cleaned.htmlis byte-for-byte identical between the two, with the same image stats and the same 18 metadata keys:Both launch paths exercised:
Headers as actually sent, captured with a local echo server:
npm run validatepasses (166 skills, exit 0).Breaking changes
None. The CLI signature, the returned JSON shape and the emitted files are unchanged.