Skip to content

fix(playwright): scope custom request headers to the navigation request - #2173

Merged
vdusek merged 5 commits into
masterfrom
fix/playwright-scope-request-headers-to-navigation
Aug 19, 2026
Merged

fix(playwright): scope custom request headers to the navigation request#2173
vdusek merged 5 commits into
masterfrom
fix/playwright-scope-request-headers-to-navigation

Conversation

@vdusek

@vdusek vdusek commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Description

  • PlaywrightCrawler._navigate() applied Request.headers page-wide via page.set_extra_http_headers(), so the browser attached them to every request the page initiated - including cross-origin subresources chosen by the crawled page. Sensitive per-request values (e.g. Authorization) could leak to third-party origins.
  • Headers are now applied through the route interceptor already used for non-GET requests, matched to the first main-frame navigation request itself rather than to its URL (a URL match breaks on browser URL normalization and on glob metacharacters in the URL). The interceptor merges the custom headers into the headers the browser sends on its own (User-Agent, fingerprint headers) instead of replacing them.
  • Behavior change: custom headers are sent with the navigation request (and inherited by its redirect hops), but no longer with any page-initiated request afterwards - subresources, XHRs, or later client-side navigations.
  • Covers PlaywrightCrawler, AdaptivePlaywrightCrawler, and StagehandCrawler, which all share _navigate(). Mirrors how crawlee JS scopes request headers in gotoExtended.
  • Added regression tests: cross-origin subresources don't receive the headers, browser-normalized URLs still do, and redirect hops inherit them.

✍️ Drafted by Claude Code

@github-actions github-actions Bot added this to the 147th sprint - Tooling team milestone Aug 18, 2026
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Aug 18, 2026
@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.54%. Comparing base (091410f) to head (0b9d79b).
⚠️ Report is 19 commits behind head on master.

Files with missing lines Patch % Lines
src/crawlee/crawlers/_playwright/_utils.py 87.50% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2173      +/-   ##
==========================================
+ Coverage   93.51%   93.54%   +0.02%     
==========================================
  Files         181      181              
  Lines       12653    12772     +119     
==========================================
+ Hits        11832    11947     +115     
- Misses        821      825       +4     
Flag Coverage Δ
unit 93.54% <90.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek
vdusek requested a review from Pijukatel August 19, 2026 07:15
Comment thread src/crawlee/crawlers/_playwright/_playwright_crawler.py Outdated
@vdusek
vdusek merged commit 6f0eff4 into master Aug 19, 2026
36 checks passed
@vdusek
vdusek deleted the fix/playwright-scope-request-headers-to-navigation branch August 19, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants