Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Question] How can I repeat a request? #16776

Closed
MsLolita opened this issue Aug 23, 2022 · 6 comments
Closed

[Question] How can I repeat a request? #16776

MsLolita opened this issue Aug 23, 2022 · 6 comments

Comments

@MsLolita
Copy link

MsLolita commented Aug 23, 2022 •
edited
Loading

What are the ways to repeat the request?
Example:

  1. I go to the page and sniff the request via route() - that is, I get cookies and headers and then make a request with them.
  2. I go to the page and then use APIRequestContext with parametres what I want.
    These two methods that I guessed about are purely crutches for my question - they don't work because I don't know exactly but need js rendering. Is it possible in the request handler to write, for example: route.repeat().
@yury-s
Copy link
Member

yury-s commented Aug 23, 2022

These two methods that I guessed about are purely crutches for my question - they don't work because I don't know exactly but need js rendering.

Can you elaborate on what exactly didn't work? Seeing you code snippet may help us better understand the problem.

Is it possible in the request handler to write, for example: route.repeat().

No.

What you described sounds like a right approach, you cannot force the page send another request but you can send one via APIRequestContext or by evaluating fetch() in the page.

Check out our Replay from HAR guide, it may work for your use case if you want to mock some requests.

@MsLolita
Copy link
Author

MsLolita commented Aug 24, 2022 •
edited
Loading

Minimum code snippet:

from playwright.sync_api import sync_playwright
import httpx

def handle_route(route):
    res = httpx.get(url, headers=route.request.headers)
    print(res.text)
    route.continue_()

def run(playwright):
    chromium = playwright.firefox
    browser = chromium.launch()
    context = browser.new_context()
    page = context.new_page()
    page.route(url, handle_route)
    page.goto(url)
    browser.close()

if __name__ == "__main__":
    url = "https://tickets.fcbayern.com/internetverkaufzweitmarkt/SelectTicketMarketTickets.aspx"
    with sync_playwright() as playwright:
        run(playwright)

response from this request:

<!DOCTYPE html>
      <html>
      <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
        <title>Challenge Validation</title>
        <link rel="stylesheet" type="text/css" href="/_sec/cp_challenge/sec-3-8.css">
        <script type="text/javascript">function cp_clge_done(){location.reload(true);}</script>
        <script src="/_sec/cp_challenge/sec-cpt-int-3-8.js" async defer></script>
        <script type="text/javascript">sessionStorage.setItem('data-duration', 5);</script>
      </head>
      <body>
        <div class="sec-container">
          <div id="sec-text-container"><iframe id="sec-text-if" class="custmsg" src="/_sec/cp_challenge/crypto_message-3-8.htm"></iframe></div>
          <div id="sec-if-container">
            <iframe id="sec-cpt-if" class="crypto" data-key="" data-duration=5 src="/_sec/cp_challenge/ak-challenge-3-8.htm"></iframe>
          </div>
        </div>
      </body>
    </html>

But in browser this response is not 50 lines but 1000 line of code. And how I can repeat it?
UPD: Node or python no matter

@MsLolita
Copy link
Author

These two methods that I guessed about are purely crutches for my question - they don't work because I don't know exactly but need js rendering.

Can you elaborate on what exactly didn't work? Seeing you code snippet may help us better understand the problem.

Is it possible in the request handler to write, for example: route.repeat().

No.

What you described sounds like a right approach, you cannot force the page send another request but you can send one via APIRequestContext or by evaluating fetch() in the page.

Check out our Replay from HAR guide, it may work for your use case if you want to mock some requests.

and how you think?

@yury-s
Copy link
Member

yury-s commented Aug 26, 2022

The response I'm getting locally is different from above. Most likely a bot detection kicks in on the server end because the request differs from what the browser sends, the intercepted request might lack some of the low-level network headers (e.g. sec-ch-* etc.). Circumventing bot detection is not a goal for playwright. You can sniff the request on the server end, compare it with the intercepted request and pass missing headers to page.request.get() manually. If you figure out what makes the server return different response or provide a reliable repro that we could run locally we can help further.

@MsLolita
Copy link
Author

The response I'm getting locally is different from above. Most likely a bot detection kicks in on the server end because the request differs from what the browser sends, the intercepted request might lack some of the low-level network headers (e.g. sec-ch-* etc.). Circumventing bot detection is not a goal for playwright. You can sniff the request on the server end, compare it with the intercepted request and pass missing headers to page.request.get() manually. If you figure out what makes the server return different response or provide a reliable repro that we could run locally we can help further.

I substituted headers and cookies from Playwright in the request, but I received a part of the code that was not loaded. As I understand it, I need to use js rendering to pass the bot anti-detection, that is, browsers do this and execute javascript code on the page and let me in. But I would like to do it via a server request.

@pavelfeldman
Copy link
Member

Closing as per above. If you don't want to perform the request outside the browser, you can always try fetching from within the page using page.evaluate. In either case this falls outside of the Playwright scope for automation and testing, hence closing.

@snyk-bot snyk-bot mentioned this issue Mar 2, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pavelfeldman @yury-s @MsLolita