Support proxies out of the box (also: potential problems with header overwrites in download handler)

[pawelmhm]

I'm trying to use scrapy-playwright with firefox and proxies and it's not easy.

In Playwright-Python and Node as well just passing proxy config to server is not enough because authorization headers are missing on request, so I need to set extra headers with page.

setting proxy for firefox in playwright (without scrapy)

Looks like this in pure playwright (no scrapy yet)

import asyncio
import logging
import os
import re
import sys

from playwright.async_api import async_playwright
from w3lib.http import basic_auth_header

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)



async def handle_request(request):
    logger.debug(f"Browser request: <"
                      f"{request.method} {request.url}>")


async def handle_response(response):
    # Log responses, just so you know what's going on when Scrapy
    # seems to be inactive
    msg = f"Browser crawled ({response.status}): "
    logger.debug(msg + response.url)
    body = await response.body()
    logger.debug(body)


async def main():
    url = 'https://httpbin.org/headers'
    CRAWLERA_APIKEY = os.environ.get('CRAWLERA_APIKEY')
    CRAWLERA_URL = os.environ.get('CRAWLERA_HOST')

    proxy_auth = basic_auth_header(CRAWLERA_APIKEY, '')
    proxy_settings = {
        "proxy": {
            "server": CRAWLERA_URL,
            "username": CRAWLERA_APIKEY,
            "password": ''
        },
        "ignore_https_errors": True
    }

    DEFAULT_HEADERS = {
        'Proxy-Authorization': proxy_auth.decode(),
        "X-Crawlera-Profile": "pass",
        "X-Crawlera-Cookies": "disable",
    }
    async with async_playwright() as p:
        browser_type = p.firefox
        timeout = 90000
        msg = f"starting rendering page with timeout {timeout}ms"
        logger.info(msg)
        # Launching new browser
        browser = await browser_type.launch()
        context = await browser.new_context(**proxy_settings)
        page = await context.new_page()

        # XXX try to run it with/without this line
        # it gives 407 without it, 200 with
        await page.set_extra_http_headers(DEFAULT_HEADERS)

        page.on('request', handle_request)
        page.on('response', handle_response)
        await page.goto(url, timeout=timeout)

asyncio.run(main())

Without setting extra headers I get this:

python proxies.py
2021-10-15 12:41:19,819 - __main__ - INFO - starting rendering page with timeout 90000ms
2021-10-15 12:41:21,123 - __main__ - DEBUG - Browser request: <GET https://httpbin.org/headers>
2021-10-15 12:41:21,707 - __main__ - DEBUG - Browser crawled (407): https://httpbin.org/headers
2021-10-15 12:41:21,713 - __main__ - DEBUG - b''

with setting extra headers I get good response and I can see traffic in my proxy logs

python proxies.py
2021-10-15 12:42:28,019 - __main__ - INFO - starting rendering page with timeout 90000ms
2021-10-15 12:42:29,549 - __main__ - DEBUG - Browser request: <GET https://httpbin.org/headers>
2021-10-15 12:42:30,594 - __main__ - DEBUG - Browser crawled (200): https://httpbin.org/headers
2021-10-15 12:42:30,597 - __main__ - DEBUG - b'{\n  "headers": {\n    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", \n    "Accept-Encoding": "gzip, deflate, br", \n    "Accept-Language": "en-US,en;q=0.5", \n    "Host": "httpbin.org", \n    "Sec-Fetch-Dest": "document", \n    "Sec-Fetch-Mode": "navigate", \n    "Sec-Fetch-Site": "cross-site", \n    "Upgrade-Insecure-Requests": "1", \n    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0", \n    "X-Amzn-Trace-Id": "Root=1-61695b16-097385cb043d01b63d71eb58"\n  }\n}\n'

setting proxy for scrapy-playwright

Now I'm trying to do the same thing in scrapy-playwright, and I run into problems.

I cannot easily set extra headers now. I can set event handlers, but request event handler according to docs does not allow modifying request object. Setting extra headers need to be done before page.go() and after context created, there is no easy way to do it in spider object now, unless I'm missing something. If I am missing something let me know.

To bypass this I subclassed downloader handler.

from scrapy import Request
from scrapy_playwright.handler import ScrapyPlaywrightDownloadHandler

from properties.settings import DEFAULT_PLAYWRIGHT_PROXY_HEADERS


class SetHeadersDownloadHandler(ScrapyPlaywrightDownloadHandler):
    async def _create_page(self, request: Request):
        page = await super()._create_page(request)
        await page.set_extra_http_headers(DEFAULT_PLAYWRIGHT_PROXY_HEADERS)
        return page

and defined it in settings. This is a hack as create_page is not meant to be modified, but it works for setting authorization. Still I get 407.

The only way I can make it work is by disabling scrapy-playwright route handler, so I comment out these lines here: https://github.com/scrapy-plugins/scrapy-playwright/blob/master/scrapy_playwright/handler.py#L151
between 151 and 161.

Now I get proper result, spider gets 200 responses, no 407 in logs and traffic going via proxy.

sugestions

• add support for proxy setting built in scrapy-playwright. To set proxy properly for different browsers users could just set PLAYWRIGHT_PROXY_HOST, PLAYWRIGHT_PROXY_USERNAME etc and scrapy-playwright will do all it needs inside download handler. I tested with firefox, but I know in chrome you may need to pass different settings to context, different browsers will have different arguments. This will be a pain to set up for most users, doing it in scrapy-playwright will make things easy for people.
• find out why await page.unroute and page.set_extra_http_headers in handler seems to interfere with each other, I don't really understand it well, I see in my log output from spider that request is made with authorization header but still proxy is responding with 407 response, need to go to make_request_handler and go step by step find out whats wrong here. I'll try to do it next week and publish my findings.

Scrapy spider code:

import logging
import os

import scrapy
from playwright.async_api import Response, Request

logger = logging.getLogger(__name__)


async def handle_response(response: Response):
    logger.info(response.url + " " + str(response.status))
    logger.info(response.headers)
    return


async def handle_request(request: Request):
    logger.info(request.headers)


CRAWLERA_APIKEY = os.environ.get('CRAWLERA_APIKEY')
CRAWLERA_URL = os.environ.get('CRAWLERA_HOST')


class SomeSpider(scrapy.Spider):
    name = 'example'
    start_urls = [
        "http://httpbin.org/headers"
    ]
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "some_project.downloader.SetHeadersDownloadHandler",
            "https": "some_project.downloader.SetHeadersDownloadHandler"
        },
        "PLAYWRIGHT_CONTEXTS": {
            1: {
                "ignore_https_errors": True,
                "proxy": {
                    "server": CRAWLERA_URL,
                    "username": CRAWLERA_APIKEY,
                    "password": "",
                }
            }
        }
    }
    default_meta = {
        "playwright": True,
        "playwright_context": 1,
        "playwright_page_event_handlers": {
            "response": handle_response,
            "request": handle_request
        }
    }

    def start_requests(self):
        for x in self.start_urls:
            yield scrapy.Request(
                x, meta=self.default_meta
            )

    def parse(self, response):
        for url in ["http://httpbin.org/get", "http://httpbin.org/ip"]:
            yield scrapy.Request(url, callback=self.hello,
                                 meta=self.default_meta)

    def hello(self, response):
        logger.debug(response.body)

[pawelmhm]

Ok, so the problem is overwriting playwright headers with scrapy headers here: https://github.com/scrapy-plugins/scrapy-playwright/blob/master/scrapy_playwright/handler.py#L218

I set headers with set_extra_headers, add authorization and proxy headers, but they are overwritten with Scrapy headers.

I wonder if this is a good idea? Why do we do it? Is it really needed?

For my use case I'm completely fine with playwright headers, I don't need all Scrapy headers. It can also lead to some unexpected problems, for example some Scrapy middleware will set some header that playwright is not expecting or not supporting, e.g. some unusual encoding not supported by playwrgith but supported by Scrapy (rare case but everything can happen).

Other potential problem it can lead to: scrapy-crawlera middleware will set authorization headers for spider that has crawlera enabled but browser context will not be started with proxy url, thus proxy credentials will leak to website.

edit:
Some work started here: https://github.com/scrapy-plugins/scrapy-playwright/compare/master...pawelmhm:proxy-support?expand=1 added mock proxy server for testing and one test for proxy without authentication for firefox, this is passing without changes.

[elacuesta]

Many thanks for the awesome research. What happens if you pass the proxy headers in the Scrapy request?

Overriding headers is done mainly for consistency, the idea is to be as close as possible to what Scrapy does and in that sense, headers coming from Scrapy (either from requests or from middlewares) should be considered. Perhaps we can allow this to be customized somehow, perhaps by taking the header processing code to a separate function and allowing to use a custom function instead.

From what I understand, in order to use proxies one needs to specify the proxy parameter in the context and also pass authorization headers. One thing that could be done would be to store the credentials from the context and add them when processing the headers. What do you think about this approach?

[pawelmhm]

proxies

From what I understand, in order to use proxies one needs to specify the proxy parameter in the context and also pass authorization headers

Yes, for most proxies yes, but not for all of them. I actually realized now, that the problem is not any proxy but one specific proxy - Smart Proxy Manager. I found this issue here: microsoft/playwright-python#443 (comment)

It looks like for mitmproxy I can just define username, password and server and pass it to browser context and it works.

Following code:

 async with async_playwright() as p:
        proxy_url = 'localhost:9001'
        browser_type = p.chromium
        proxy_settings = {
            "proxy": {
                "server": proxy_url,
                "username": "pawel",
                "password": "yes"
            }
        }

        browser = await browser_type.launch(
            args=[
                 '--ignore-certificate-errors',
                 '--no-sandbox',
                 "--disable-web-security"
            ],
            **proxy_settings
        )

work when I try with authenticated mitmproxy, I launch it like this

> mitmproxy -p 9001 --proxyauth "pawel:yes"

and requests are fine.

But when I try same code with Smart Proxy Manager it fails. I think we should just notify owners of Smart Proxy Manager about this, it looks like a bug and there should be some solution for this in SPM.

Setting authorization headers with set_extra_headers call is not supported by chromium, because of this, author of this blog post described the problem, but his solution is not possible with playwright as there is no page.authenticate method.

To conclude, proxy issue should not be about playwright or scrapy-playwright, but I think it should be done by better support by Smart Proxy Manager for headless browser.

headers

Overriding headers is done mainly for consistency, the idea is to be as close as possible to what Scrapy does and in that sense, headers coming from Scrapy (either from requests or from middlewares) should be considered.

I think the approach of doing the same as Scrapy is problematic, as browser will not do the same thing as Scrapy, it will render differently. Browser vendors are adding headers intentionally, and here we are modifying their design by replacing their headers with different set of headers from different downloading software such as Scrapy. Also most middlewares from Scrapy will not think that their headers will be used by playwright rendering, users creating middlewares will probably assume that headers they add will work with Scrapy downloader and not with playwright.

There is also a problem of fingerprinting. There are different use cases, but some people will use headless browser precisely to avoid showing Scrapy fingerprint to website. Overriding headers will show Scrapy fingerprint and not browser fingerprint.

On the other hand using Scrapy headers may be more 'ethical' and fair as it shows to website tha it is not dealing with real browser but with Scrapy. But here we could do the same thing by just adding something to User-Agent by default, for example something like

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/75.0.3738.0 Safari/537.36 Scrapy 2.5.1

so adding Scrapy at the end of User-Agent string.

To conclude I would add some customizations to allow NOT to override headers, or maybe even make not overriding headers the default behavior and add some switch for people to enable it if they need it.

[elacuesta]

I don't know exactly which upstream changes were made or when, but SPM has been working for some time with the instructions at https://github.com/scrapy-plugins/scrapy-playwright#proxy-support.
Thanks again for the research Paweł!