[Question] inside a page.response or page.requestcompleted handler i can't get the page body. It is a bug ?

[MauroMFerreira]

...
def request_finished_handler(request):
        response = request.response()
        ...
        if request.redirect_to==None and request.resource_type in [ 'document','script' ]:
            body = response.body()    # error => Response body is unavailable for redirect responses
            body = response.text()      # error => Response body is unavailable for redirect responses
            body = page.content()      # error => Execution context was destroyed, most likely because of a navigation.
            body = page.evaluate('document.body') # error => Execution context was destroyed, most likely because of a navigation.
 ...
page.on("requestfinished", request_finished_handler)
page.goto(url)

I also tried with:

def response_handler(response):
        request = response.request
...
page.on("response", response_handler)
page.goto(url)

Problem is, I don't need the body of the final page loaded, but the full bodies of the documents and scripts from the starting url until the last link before the final url, to learn and later avoid or spoof fingerprinting.

I can - and i am using by now - requests.get() to get those bodies, but this have a major problem: being outside playwright, can be detected and denied as a scrapper (no session, no referrer, etc.), so i want to avoid this hack.

It is a bug or there is a way to do this that i don't know ?

Thanks !

[mxschmitt]

It's expected, that there is no body or text when its a redirect. You can detect it based on the response status code. Your use-case seems not that clear, if its only about the response bodies, you can already do it today and it works see here:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.on("requestfinished", lambda req: print(req.url, req.resource_type, req.response().body()))
    page.goto('http://whatsmyuseragent.org/')
    page.wait_for_timeout(1000)
    browser.close()

The target, closed errors you get, because you are trying to get the body, which is internally a request to the browser but you already closed the page, context, or browser so it gets canceled.

If you have a concrete snippet of whats not working, let us know!

[MauroMFerreira]

@mxschmitt:

1. "It's expected, that there is no body or text when its a redirect.". Yes, that's why the "if request.redirect_to==None and request.resource_type in [ 'document','script' ]:". Could be request.status>299 and request.status<400, but the result will be poorer;

2. Your code just give the final page; i explained that's it's not what i want: "Problem is, I don't need the body of the final page loaded, but the full bodies of the documents and scripts from the starting url until the last link before the final url, to learn and later avoid or spoof fingerprinting".
   And so i'm using a page.requestcompleted (or page.response, but with the same results, and page.request and page.route don't do anything usefull for me) handler to try to get the deep link bodies that are redirects of type meta_equiv, location_href, location_assign, location_replace and cases of links a_href that are 'clicked' by js scripts: all of those redirections are made in the browser, so they need to have a body, and the browsers must load and run those bodies to act and do those redirections. Problem is, playwright act as they don't exists.

Mauro

[mxschmitt]

My code will also list all the sub-resources of the page, including scripts, styles, fonts etc. Could you elaborate what the "starting URL" and the "last link before the final url" is in your scenario?

Page.route is mostly for request interception, thats nothing which you need in your case I guess. It seems like the Playwright layer is the not the right tool for your use-case. Maybe the Chromium extension API gives you more flexibility there - but just a wild guess, since the scenario in terms of what it has to do with fingerprinting is not clear to me.

Closing since its not about Playwright anymore.

[MauroMFerreira]

Ok. Have it your way.