Reduce allocations made during request interception

[karlseguin]

Stream (to json) the Transfer as a request and response object in the various network interception-related events (e.g. Network.responseReceived).

Add a page.request_intercepted boolean flag for CDP to signal the page that requests have been intercepted, allowing Page.wait to prioritize intercept handling (or, at least, not block it).

[karlseguin]

@krichprollsch , I'd like your input on this. I don't think this is a new issue, just more easily observable with request interception.

We currently emit a CDP Page.frameStoppedLoading event at the same time that we emit the JavaScript 'load' event, i.e when all non-async scripts are completed. I believe this is the event that playwright uses to unblock a page.goto($URL) call. Because of where/how we emit this, XHR requests can be inflight when Page.frameStoppedLoading is emitted.

I think this is different than Chrome, which appears to wait until XHR requests are complete. Although, looking at the Chromium source code, some tests mention/handle possible race conditions between XHR requests and the Page.frameStoppedLoading event.

This comes up because, if you do something:

const page = await context.newPage();
await page.route('**/*', async (route, request) => {
  setTimeout(() => route.continue(), 2000); // SLEEPING HERE
});
await page.goto('/campfire-commerce/');

await page.close();

It'll definitely call page.close() before the XHR requests are issued. In Chrome, it will wait until the XHR requests are complete.

I see two options:

1 - We can change when Page.frameStoppedLoading is called, adding another internal event that happens after .page_navigated when all (known??) xhr requests are complete.

2 - We can consider the above playwright script unsupported. After goto returns and before close is called, the script should query the document state, e.g.:

await page.goto('/campfire-commerce/');

// add this
await page.waitForFunction(() => {
    const price = document.querySelector('#product-price');
    return price.textContent.length > 0;
}, {}, {});

await page.close();

[krichprollsch]

Well, sending the frameStoppedLoading after xhr calls sounds appealing, but I'm really not sure it is correct.

I used crush to look at the chromium code, and it said clearly the frameStoppedLoading is not related to xhr calls, it doesn't wait for any network request.

Does the puppeter example you wrote acts differently with chrome?

---

Is frameStoppedLoading dispatched just after the onload event?

Yes, frameStoppedLoading is dispatched very shortly after the onload event, but the exact timing can vary
slightly.

The sequence is:

1. Document parsing and resource loading completes
2. Document ready state changes to "complete" ( SetReadyState(kComplete) )
3. onload event is dispatched via Document::ImplicitClose() → LocalDOMWindow::DocumentWasClosed() →
   LocalDOMWindow::DispatchLoadEvent()
4. frameStoppedLoading is dispatched via ProgressTracker::ProgressCompleted() →
   probe::FrameStoppedLoading(frame_)

Key findings:

1. Same call chain: Both events are triggered from the same document completion flow in
   Document::CheckCompletedInternal()
2. Close timing: The frameStoppedLoading event is sent from ProgressTracker::ProgressCompleted() which is called
   from FrameLoader::DidFinishNavigation() very shortly after the load event
3. Race condition with XHR: As confirmed by the test comment, XHR requests initiated in the onload handler can race
   with frameStoppedLoading because:
   • The onload event fires first
   • JavaScript in the onload handler can start XHR requests
   • frameStoppedLoading fires shortly after, potentially before those XHR requests complete
4. Asynchronous dispatch: In some cases, the onload event may be dispatched asynchronously via PostTask() , which
   could affect the exact timing

Practical implications:

• frameStoppedLoading indicates the initial document load is complete
• It does not wait for JavaScript-initiated requests (XHR, fetch, dynamic resources)
• If you need to wait for all network activity, monitor individual network requests via the Network domain instead

The events are very close in timing, but frameStoppedLoading should generally come after onload , though the exact
order may vary due to asynchronous task scheduling.

---

Based on my comprehensive analysis of the Chromium codebase, I can now provide a detailed answer about what async
tasks frameStoppedLoading waits for.

What async tasks frameStoppedLoading waits for:

The frameStoppedLoading event waits for specific types of load-blocking resources and operations, but NOT for
arbitrary async JavaScript tasks like XHR/fetch. Here's what it waits for:

1. Load-blocking resource types:

• Images ( ResourceType::kImage )
• CSS stylesheets ( ResourceType::kCSSStyleSheet )
• Fonts ( ResourceType::kFont )
• SVG documents ( ResourceType::kSVGDocument )
• XSL stylesheets ( ResourceType::kXSLStyleSheet )

2. Document parsing completion:

• parsing_state_ == kFinishedParsing - The HTML parser must be finished

3. Load event delay counters:

• Elements that explicitly delay the load event via IncrementLoadEventDelayCount() :
• Script elements (handled separately in ScriptRunner)
• Media elements ( video , audio ) during certain loading phases
• Plugin elements ( embed , object )
• Style elements during stylesheet loading
• Link elements during stylesheet/resource loading
• SVG elements during resource resolution
• Processing instructions (like XSL)

4. Child frame loading:

• AllDescendantsAreComplete() - All child frames (iframes) must finish loading

5. Active resource requests:

• !Fetcher()->IsInRequestResource() - No resource requests currently being processed
• !fetcher_->BlockingRequestCount() - No blocking resource loaders active

6. JavaScript URL tasks:

• !javascript_url_task_handle_.IsActive() - No pending javascript: URL execution

What it does NOT wait for:

Non-blocking resource types:

• Scripts ( ResourceType::kScript ) - handled separately
• XHR/Fetch requests ( ResourceType::kRaw )
• Prefetch requests ( ResourceType::kLinkPrefetch )
• Text tracks ( ResourceType::kTextTrack )
• Audio/Video ( ResourceType::kAudio , ResourceType::kVideo )
• Web App Manifests ( ResourceType::kManifest )
• Speculation rules ( ResourceType::kSpeculationRules )

JavaScript-initiated async operations:

• XMLHttpRequest calls
• fetch() API calls
• setTimeout / setInterval callbacks
• Promise chains
• Dynamic imports
• Service Worker operations
• IndexedDB operations

Summary:

frameStoppedLoading waits for the initial document and its critical rendering resources to load, but not for
JavaScript-initiated async operations. It's designed to indicate when the page's initial loading phase is complete,
not when all possible async activity has finished.

This is why XHR requests in onload handlers can race with frameStoppedLoading - the event fires when the
document's initial load is complete, regardless of what JavaScript does afterward.

[karlseguin]

@krichprollsch I thought I was seeing that, but I checked again, and it and does not wait for XHR requests. I'm not sure what I thought I saw now. thanks