Cannot Overwrite Headers for Python's `Route.continue_`

[flynnai]

Hi all! I first want to say that I love Playwright -- it is incredibly seamless and lightning fast. From my experience so far, it has a simple yet powerful API, and is a lot less clunky than Selenium.

My team is trying to use it to render PDFs, and we need to set the Origin header to avoid CORS issues. I have found an issue that has plagued me on Ubuntu and MacOS Big Sur, with Python 3.9.5 and Playwright (Python) 1.12.1.

Using a modified Spec example, here is our issue:

from playwright.sync_api import sync_playwright

def handle(route, request):
    headers = {
        **request.headers,
        'foo': 'bar', # set 'foo' header
        'origin': 'literally_anywhere'
    }
    route.continue_(headers=headers)

with sync_playwright() as pw:
    browser = pw.chromium.launch()
    page = browser.new_page()
    page.route("**/*", handle)
    page.on('request', lambda request: print(f'====request headers: ({request.url})====\n{request.headers}\n'))
    browser.close()

'foo' is being set erratically -- only on two of the page requests that are logged. 'origin' is never set.
In addition, the spec example cites being able to set 'origin': None, but for me it throws

playwright._impl._api_types.Error: headers[4].value: expected string, got object

Installation procedure was as follows:

# (in a python3 venv virtual environment)
pip install playwright
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 python -m playwright install
playwright install chromium

Thank you in advance for any help! Please let me know if this is just a configuration issue.

[mxschmitt]

I unfortunately can't reproduce. Also in your reproducible, you never make a http request or navigate, so the handler gets never executed. I tried it with a manual goto which does not cause any issues and the new request headers are set correctly.

PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 python -m playwright install is like disabling browser downloads but calling the CLI to install it, do you have your own Chromium browsers somewhere which you are using via the executable path?

I've used the following test-case:

async def test_todo(
    page, server
):
    await page.route(
        "**/*",
        lambda route, request: route.continue_(headers={**request.headers, "foo": "bar",
        'foo': 'bar', # set 'foo' header
        'origin': 'literally_anywhere'}),
    )

    for _ in range(10):
        response = await page.goto(server.EMPTY_PAGE)
        assert response.request.headers["foo"] == "bar"
        assert response.request.headers["origin"] == "literally_anywhere"
        print(response.request.headers)

[flynnai]

Oops, I had removed the goto in my original post on accident when I slimmed the code down for the sake of example. Oddly enough, it's still not working for me:

import asyncio
from playwright.async_api import async_playwright

async def test_todo(
    page
):
    await page.route(
        "**/*",
        lambda route, request: route.continue_(headers={**request.headers, "foo": "bar",
        'foo': 'bar', # set 'foo' header
        'origin': 'literally_anywhere'}),
    )

    for _ in range(10):
        response = await page.goto("http://google.com")
        assert response.request.headers["foo"] == "bar"
        assert response.request.headers["origin"] == "literally_anywhere"
        print(response.request.headers)


async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await test_todo(page)
        await browser.close()

asyncio.run(main())

Gives me

  File "/Users/aidanflynn/Desktop/testing/playwright/other.py", line 17, in test_todo
    assert response.request.headers["origin"] == "literally_anywhere"
AssertionError

PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 python -m playwright install is like disabling browser downloads but calling the CLI to install it, do you have your own Chromium browsers somewhere which you are using via the executable path?

I'm not sure - on my Mac I have Chrome installed. Oddly enough, a quick test confirms that running

python3 -m venv env
source env/bin/activate
pip install playwright
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 python -m playwright install

(without ever installing chromium) is enough to get access to chromium (I can access pw.chromium.launch() and everything, and take a PDF capture of google.com). Does that help?

But I used the above commands, plus playwright install chromium on Debian (which never had Chrome installed beforehand), and it still wasn't working.

[mxschmitt]

I digged into it. The reason is at least in the reproducible, that it's based on redirects which is currently an upstream limitation. When you are using https://www.google.com it works. Their redirect chain is:
http://google.com -> https://google.com -> https://www.google.com.

PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 python -m playwright install this will like result in doing nothing. You are saying basically, hey install browsers but the skip env var is set, so its not installing anything at all.

[pavelfeldman]

Sorry, this thread is now tricky to follow. Is the issue with route not overriding the headers? Or that it prints original headers? It will print originals, but will send modified over the network. So your original report looks like "works as intended". But the thread seems to be changing the issue at hand, so I don't know if your issue is resolved.

[mxschmitt]

Closed for now, lets reopen if the problem is still persistent.