[Feature] Capture network requests on BrowserContext

[vigneshshanmugam]

Currently when page issues requests, we can use the request event on the page to capture the network request and get the timing information. But if the page opens a new popup or window, the network request information for those requests gets lost

<p><a href="popup2.html" target="_blank" id="linkWithPopupNoNewWindow">Click</a>here (id=linkWithPopupNoNewWindow) to open - Test Page - new tab</p>

// will not log the requests to the popup page and also the resources in page1. 
page.on("request", req => console.log("Request", request.url());

await page.goto("index.html"); // goes to index page

const [page1] = await Promise.all([
      context.waitForEvent('page')
      page.click('#linkWithPopupNoNewWindow'),
]);

await page1.close();

From the above code, we would miss out the logging for both the request to the popup2.html page and also the resources inside that page. However, the resources inside page1 can be logged by again listening to page1.on('request'), but still there is no way to get the request activity to the popup in new tab.

Is there a way to get all the network activity logged in the context of the browser level CDP session instead of creating CDP session for every page object.

[dgozman]

We can do something like browserContext.on('request') that captures requests from all the pages in the context.

[vigneshshanmugam]

@dgozman Yeah that would be really cool for the logging case. Would it make sense to have the context.newCDPSession function that does not take page as argument and instead just creates a global CDP session for all targets that gets created across the playwright browser execution?

    const client = await context.newCDPSession();
    client.on('Network.RequestWillBeSent', console.log)
    await client.send('Network.enable');

[dgozman]

@vigneshshanmugam CDP works per-target, so such an API on the context will not match the internals and I don't think we should do something like this. Note that you can use newBrowserCDPSession() and manually attach to all pages inside it.

If you are willing to contribute browserContext.on('request'), that would be wonderful!

[vigneshshanmugam]

CDP works per-target, so such an API on the context will not match the internals and I don't think we should do something like this. Note that you can use newBrowserCDPSession() and manually attach to all pages inside it.

Gotcha, that makes sense. I tried working through newBrowserCDPSession, is it possible to use the playwright API instead of manually dealing with browser targets and creating sessions for each target - page creation? is there a better way to create the session using the playwright API using the targetinfo.sessionid.

  const client = await browser.newBrowserCDPSession();
  client.on('Target.targetCreated', async payload => {
      if (payload.targetInfo.type == 'page') {
        console.log('Created page', payload.targetInfo.url);
        // const pages = await this.context.pages();
        // for (const page of pages) {
        //   const session = await this.context.newCDPSession(page);
        //   await this.intercept(session);
        // }
      }
    });

If you are willing to contribute browserContext.on('request'), that would be wonderful!

Totally, would like to take it for a spin.

[dgozman]

Gotcha, that makes sense. I tried working through newBrowserCDPSession, is it possible to use the playwright API instead of manually dealing with browser targets and creating sessions for each target - page creation? is there a better way to create the session using the playwright API using the targetinfo.sessionid.

Not really. We expose CDPSession for those who feel adventurous or have some tooling already built upon CDP. It is not cross-browser-compatible, so we are unlikely to build APIs on top of it. Instead, we prioritize features that help many users and implement them across all browsers.

Totally, would like to take it for a spin.

Great! I'd start with src/client/events.ts and look around for the 'request' event. Let me know if you have any questions - happy to help.

[vigneshshanmugam]

We expose CDPSession for those who feel adventurous or have some tooling already built upon CDP. It is not cross-browser-compatible, so we are unlikely to build APIs on top of it.

Yeah can reason to that one. One of the reasons why we went with using CDP is because the page.on(request) event does get all of the details as per https://chromedevtools.github.io/devtools-protocol/tot/Network/#event-requestWillBeSent. Reason being we need those details to get the blocking, queued and other timing info required to construct the waterfall appropriately which is what request event lack and went with CDP instead. Is there a possibility to expose that information at some point?

Great! I'd start with src/client/events.ts and look around for the 'request' event. Let me know if you have any questions - happy to help.

Sure, thanks 👍

[vigneshshanmugam]

Is there a possibility to expose that information at some point?

It would be really cool, if playwright can expose the original unmodified event as second argument to the request event which would eliminate the extra browser session handling that we discussed before. Happy to do a PR as well. Let me know if would work for the playwright team

 page.on('request', (request: Request, event: Protocol.Network.requestWillBeSentPayload) => {
        // code for working with raw events from CDP
 })
// same for `requestfailed` and `requestfinished`

[dgozman]

It would be really cool, if playwright can expose the original unmodified event as second argument to the request event which would eliminate the extra browser session handling that we discussed before. Happy to do a PR as well. Let me know if would work for the playwright team

This is not going to work, because it's not cross-browser. However, we can expose additional information if you need something, directly on the Request object. Please file a separate issue for that, and we'll triage/prioritize it accordingly.

[vigneshshanmugam]

his is not going to work, because it's not cross-browser. However, we can expose additional information if you need something, directly on the Request object. Please file a separate issue for that, and we'll triage/prioritize it accordingly.

sure, will create a separate issue to expose some more information from the raw events.