[Question] Is it possible to make an HttpRequest through Playwright?

[symon-skelly]

I currently use Cypress for my automation testing framework, however I am toying with the idea of moving over to Playwright. One feature of cypress that really shines in the ability to make HttpRequests using the cy.request() function, I use this many times throughout the current system however the most important request is used to setup my randomly generated user application state before I even visit the web-app itself (it makes the request to our webServer which responds with data that is then used for the user-specific localStorage items), this allows me to skip the login screen of my application entirely, saving a lot of time for each test.

I have read the docs about how Playwright can re-use application state but this is not really a viable solution for me because after each test I specifically use new application state.

TLDR: Is it possible to make an HttpRequest through Playwright and get its response?

[mxschmitt]

You can use the Fetch API inside the browser to make http requests. Example for Playwright:

// @ts-check
const playwright = require('playwright');

(async () => {
  const browser = await playwright.chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('http://example.com');
  const response = await page.evaluate(async () => {
    return await fetch("http://worldtimeapi.org/api/timezone")
      .then(r => r.ok ? r.json() : Promise.reject(r))
  })
  console.log(response)
  await browser.close();
})();

Inside of evaluate you can access the LocalStorage etc. To pass arguments over to the browser process, you can pass it to page.evaluate() in the second argument. See here for the docs.

[symon-skelly]

That looks like a solution that could work for me, thank you for your fast reply!

[pavelfeldman]

It looks like what you really want is this: https://playwright.dev/docs/auth#reuse-authentication-state. You can even create this state using cli once: https://playwright.dev/python/docs/cli#preserve-authenticated-state

[pavelfeldman]

Now that I read you request to the end, I see that you use different state for each test. Is it something like incremental session id that the server keeps tracking of? Figuring out if we can get something out of the box for all the auth scenarios...

[shirshak55]

I really don't like this fetch system.

Issue with fetch

• Getting raw bytes etc,
• cors issue (sometime even have to disable security feature) etc.

For now what I do is grab all cookies and send it to Got or request promise. I wish there are really good way to send request.

[andrisi]

It would be useful in other cases too, making request on behalf of the running web app (using it's state and cookies and all), and a simple API that uses best practices and universal among browsers would be great. Thanks.

[yury-s]

Issue with fetch

• Getting raw bytes etc,
• cors issue (sometime even have to disable security feature) etc.

Would simple http.request() with corresponding headers populated from the context work or there is something else that it lacks?

The following snippet would not be a subject to cors and will give access to the raw bytes. @symon-skelly would something like this work for your scenarios?

  // Copy cookies from the context
  const cookies = await context.cookies('http://localhost:8907');
  const cookiesHeader = cookies.map(c => `${c.name}=${c.value}`);

  const response = await new Promise(fulfill => {
    const http = require('http');
    const req = http.request({
      hostname: 'localhost',
      port: 8907,
      path: '/empty.html',
      method: 'GET',
    }, res => {
      fulfill(res);
    });
    req.setHeader('Cookie', cookiesHeader);
    req.end();
  })
  // Handle response here...
  console.log(`got response: ${response.statusCode}`);

[shirshak55]

@yury-s tbh no. There are problems with using got etc.

1. Too complicated than native fetch api.
2. Cookies can change between request. Now people have to use cookie jar etc.
3. Chrome networking stack is different so it has different tls fingerprint.
4. I noticed sometime for getting cookies we have to visit the website and some cookies especially session cookies are not avail till we visit website.

I think just disabling the cors and using fetch would fix the issue. But that makes whole browser unsafe. In firefox we can't even disable cors which is problem in my opinion

[yury-s]

@shirshak55 thanks for your reply, still trying to better understand your requirements

2. Cookies can change between request. Now people have to use cookie jar etc.

Yeah, you have to call context.cookies on the current status of the context. This all could be encapsulated in a helper function but the idea is the same - you work with the web page and at some point use its cookies to send a custom request from playwright.

3. Chrome networking stack is different so it has different tls fingerprint.

Ok, this is interesting. cy.request doesn't use Chrome stack either and if we implemented such request API in Node it would suffer from the same problem. Is there a testing scenario where this kind of differences appear or it's scraping?

4. I noticed sometime for getting cookies we have to visit the website and some cookies especially session cookies are not avail till we visit website.

How would request API help with this? My understanding is that you'd still have to visit the site before being able to use the request API.

I think just disabling the cors and using fetch would fix the issue. But that makes whole browser unsafe. In firefox we can't even disable cors which is problem in my opinion

Playwright already has bypassCSP option that could be used to disable cors. As for the safety - in testing (which is our primary goal) it should be less of a problem since you usually run tests in a trusted/hermetic environment.

[shirshak55]

@yury-s Thanks.

For tls it not just scraping but for other uses where banks etc uses tls fingerprinting to stop request as a part of mitigation. But you are right cy will also not work.

The main reason I like using browser over node is I can debug it easily. I can see the request response in chrome developer tools network tab while making the program. With Node I have to use some mitm proxies setup certs etc which gets tedious. And managing cookie jar is not that easy to be honest. Many websites changes cookies with every request and this causes problem tbh.

And even we cannot make cookie jar easily afaik. I have tried tough cookie to make jar but till today I have not been able to use it successfully with got (another node js client I prefer).

IDK bypassing cors was available I am so happy to know such option exists. I hope it works for firefox and my problem would be solve easily. In past I did use fetch and it works fine on chrome with cors disabled but firefox doesn't work. https://github.com/shirshak55/scrapper-tools/blob/master/src/browserRequest.ts#L33

[Dospios]

I really think that Playwright needs its own analogue of cy. request. This is one of the most convenient things for cypress, despite the limitations. I'd really like to see playwright be able to send requests without all these workarounds.

[mxschmitt]

@Dospios we are curious to find more out about the use-cases why people use cy.request. What kind of requests are you making and when? Do you automate e.g. the login via the request instead of doing it via automation?

[Dospios]

@mxschmitt Yes, this is one of the use cases. If I have already written a test that checks authorization on the frontend, then for the following tests that require authorization, there is no need to perform authorization on the frontend, it is much more efficient and faster to register accounts and perform authorization on the backend. In the same way, I check third-party services, for example, stripe. It is much more convenient to make a subscription on the backend than to make a new subscription on the frontend every time. Another example: managing test data, for example, deleting a user on the backend after the test.
Here is a typical case: Create a random user on the backend => check some functionality => delete this user on the backend. This allows you to isolate the tests from each other as much as possible and at the same time effectively manage the test data with just one team.
At the moment I am writing autotests for two projects, I use cypress on one and Playwright on the other, so I involuntarily start comparing these tools, each has pros and cons. Playwright seemed to me a much more flexible tool and I like it more, but at the same time it is more convenient to work with the backend in cypress because of cy.request.

[mxschmitt]

Sounds like a valid use-case. Are you depending on making the requests in the browser? (so that the cookies are set) If not, you should be able to use e.g. node-fetch to make http requests inside your tests.

[vdhpieter]

Lovely to see this get's actively worked on. Can we somewhere follow what is still left to do?

[vdhpieter]

@yury-s @mxschmitt @pavelfeldman Any progress updates on this?

[mxschmitt]

@vdhpieter https://youtu.be/6RwzsDeEj7Y?t=265

[szmarczak]

FYI there's https://github.com/apify/got-scraping that mimics the browser fingerprint as closely as possible (TLS, headers casing, headers order etc.). There are some issues with TypeScript however it's strictly a Got issue.

Cookies can change between request. Now people have to use cookie jar etc.

This shouldn't be a problem since you can just do:

const instance = gotScraping.extend({
	cookieJar: new CookieJar(),
});

const response = await instance('https://example.com');

---

There are still some limitations related to the Node.js HTTP parser, so going with playwright may be your best bet in some use cases.

[mxschmitt]

Closing since we're releasing it with v1.16. See here: https://playwright.dev/docs/next/api-testing

[andrisi]

@mxschmitt perhaps you could call it .fetch() instead of ._request() on the Page & BrowserContext to avoid confusion with browser request events. Its a FetchRequest after all. Thanks for the great work!

[yury-s]

We debated that among other options and decided to stick with page.request.get() and context.request.get(). There is no other request field on those classes so it should not cause any problems.

[epicwhale]

I had few clarification questions around using .fetch(...) or .get(..) on the page.request APIRequestContext object.

1. It does not seem to recreate all the headers similar to the request if made from the browser's console directly (for example using fetch in javascript). Is this working as intended?

2. It does not appear in the chrome developer tools network tab... so it's hard to debug if there's a different in the fingerprint? how do I inspect the request being made from the APIRequestContext?

3. Would it also honor the proxy set up on the browser context?

[mxschmitt]

1. This is a best-effort approach, Sec headers e.g. don't fall under this. If you miss a header which is critical for you, I recommend filing a new issue. Then we can look into it.
2. The requests which are made via request.* are made via Node.js, not via the browser, they then don't end up in DevTools, this is expected. We recommend e.g. setting pw:api env var to debug them or using our Trace Viewer.
3. They will (should) honor the proxy settings on the context. If it does not work for you, I recommend filing a separate issue.

If you experience any issues, please file a new issue, since we usually don't monitor closed/stale issues. Thanks!

[epicwhale]

Thanks a ton @mxschmitt - that clarifies most of it, and I will probe further and file new requests as I progress!