Please add the initiator information

[thewulf00]

Requests, that are fired after the basic request is done, are initiated by some source - f.e. a script or something. Since chrome does provide this information in the network tab, I wonder if you can give it via puppeteer?

Example:

[jawadst]

+1, that'd be very helpful

[zicongxie]

@aslushnikov Please! That is useful for me.

[thewulf00]

Thank you for your replies. I hoped that somebody will need it too.
@aslushnikov Is it possible to get this? I find puppeteer very useful!

[Garbee]

Out of curiosity, could someone provide an example use-case for needing this information through puppeteer?

[ctavan]

@Garbee assuming you use puppeteer to analyze a website which includes 3rd-party code like social sharing buttons. You may want to know which requests were initiated by your own 1st-party code and which requests were initiated by 3rd-party code. This could help you spot performance problems like 3rd-party widgets that cause a lot of subsequent requests.

[sebastianneubert]

@Garbee Same here. We want to build request maps to vizualise which request was called from where. So we can check, where are performance/memory leaks at 3rd party includes. This could be very helpful to point out, which scripts or iframes initate a waste amount of requests/cookies/trackers. We have all information to build this, except of the initiator of the request.

[Laywanger]

+1 I need this to track malvertising
@aslushnikov Is it possible ?

[aslushnikov]

We don't currently have initiator information in the request interception events.

Filed upstream: crbug.com/800623

[joshgoodwin]

I believe that initiator is available to the event object (event.initiator) in NetworkManager.js here:

https://github.com/GoogleChrome/puppeteer/blob/master/lib/NetworkManager.js#L209

You should be able to just forward that property along like you do with event.requestId, no?

[aslushnikov]

@joshgoodwin initiator is not available for intercepted requests; that's what the upstream bug is about.

https://github.com/GoogleChrome/puppeteer/blob/8c930843160d7fff76a82bf154f151e2bdd29383/lib/NetworkManager.js#L127-L133

[sebastianneubert]

Any news about the refactoring of #1817 and the initiator information in an intercepted request?

[madecki]

It would be great to get information about cookie and request initiators!

[phackt]

Hello, any news about the initiator attribute ? thanks

[sebastianneubert]

Not really as initiator. But in the last updates of puppeteer, most requests have the referer information in the header, which is fine for me.

page.on('request', interceptedRequest => {
    const url = interceptedRequest.url();
    header_referer = interceptedRequest.headers()['referer'];
    console.log(interceptedRequest);
    interceptedRequest.continue();
});

[phackt]

@sebastianneubert yes not really the same, you don't have the name of the exact resource involved in the request, also no referer is sent with XHR. The field initiator is really missing.

[rylyade1]

@phackt actually referrer is also sent with XHR, but there is no guarantee of this header being present (e.g when the referrer policy is set to 'no-referrer'). In these cases @sebastianneubert your solution will probably throw an exception.

[nwingert]

FYI, initiator info is available using CDP directly. Here's an example from the solution I'm using atm.

// ChromeDevTools/devtools-protocol#56
// increase CallStackDepth to fix initiator stack issue in chrome
const client = await testPage.target().createCDPSession();
await client.send('Debugger.enable');
await client.send('Debugger.setAsyncCallStackDepth', { maxDepth: 32 });

// #1395
// puppeteer request events don't propagate initiator currently
// so register for the event directly with CDP instead
await client.send('Network.enable');
await client.on('Network.requestWillBeSent', (params) => {
const reqURL = params.request.url;
logger.info(request to: ${reqURL});
logger.info(from initiator: ${getInitiatorUrl(params.initiator)});
});

getInitiatorUrl is a little recursive function to search the initiator callstack for the info I need, not included because it's too specific to my requirements.

[aslushnikov]

To everybody: do I get it right that you guys need both initiator type and a stacktrace?

[ctavan]

@aslushnikov speaking for me: yes, both would be great!

This patch worked for me (kind of): ctavan@8cdda16

With this initiator() information at hand I used code like this:

    const initiator = request.initiator();
    let simplifiedInitiator;
    if (initiator.type === 'parser') {
      simplifiedInitiator = {
        type: 'parser',
        url: initiator.url
      };
    } else if (initiator.type === 'script') {
      simplifiedInitiator = {
        type: 'script',
        urls: [...(new Set(initiator.stack.callFrames.map(frame => frame.url).filter(v => Boolean(v))))],
        functions: [...(new Set(initiator.stack.callFrames.map(frame => frame.functionName).filter(v => Boolean(v))))],
      };
    } else {
      simplifiedInitiator = initiator;
    }

to extract a "simplified" version of the initiator.

[FranklinYu]

Note to myself (hope this helps someone else): the type definition of initiator is at

https://chromedevtools.github.io/devtools-protocol/tot/Network#type-Initiator

[Marmeladenbrot]

Any news about that / solution for Puppeteer to get the initiator?

[Vasile-Peste]

Is there a way to get a initiator also for cookies?

[ojaswa1942]

@Vasile-Peste +1. Looking for this too! Did you find any solution or workaround to this?

Currently the only solution I can think is to create a map between various existing parameters from requests and cookies, but that is likely to give false positives as per my use case (due to possible variations in domain and path properties of cookies).

But even then, a mapping solution with requests will still not work for cookies set using document.cookie.

[HarshiShah]

Any plans on adding the initiator information? I see pull requests for this feature open for more than 2 years.

[whalderman]

Thank you @jschfflr !

[BennyAlex]

@jschfflr

Hello,

Is it also possible to get the whole request Initiator chain?
Thanks!

[ojaswa1942]

@BennyAlex Did you happen to extract this information?

[BennyAlex]

@ojaswa1942
No unfortunatelly not :/

[OrKoN]

Looks like it's been implemented. I believe you would only get the initiator for the initial request in a redirect chain. Please file a new feature request for this.