Document overriding user-agent

[pawelmhm]

User agent from playwright is overwritten with Scrapy user agent: https://github.com/scrapy-plugins/scrapy-playwright/blob/master/scrapy_playwright/handler.py#L213 I think this should be mentioned in documentation, many users are settings browser-like user-agents in scrapy settings, e.g. Chrome user agent. Many will not think about it but from the point of view of some websites doing request fingerprinting it can look like user-agent spoofing, because they will for example use playwright Firefox, fingerprint will be for firefox, but with Chrome user-agent from settings.py

[scrapenetwork]

I am having an issue when running headless the user agent isn’t respected.

placed in settings, in spider, & in headers , works but sniffing the request shows headless/Chrome seems to always be detected and used

[Gallaecio]

@scrapenetwork What is the user-agent header sent to the server? This topic is about the header sent, not how other aspects of Playwright may be used to detect the user agent as being a headless browser despite the user agent header.

[scrapenetwork]

The Headless/chrome.
understood, appreciate the response