Provide more request headers when intercepting requests

[mattzeunert]

When a request from the example code below is intercepted, the following request headers are available (same as in requestWillBeSent):

referer, user-agent, x-custom-header

But these are the full request headers that Chrome uses when making the request:

Referer, User-Agent, x-custom-header, Accept-Encoding, Host, Accept, Cookie, Connection

Some of those depend on how the actual request is made by Chrome, but these don't:

• Cookie (currently I'm generating this by using the cookie API)
• Accept (it's probably possible to find a sensible value for this based on the request type)

My end goal would be that I can make the request myself and it will have the same end result as if Chrome had made the request. (So even headers like Accept-Encoding would be helpful.)

The specific problem I've been debugging is that the backend I'm working with was sending different response content based on the value of the Accept header.

---

Example code:

var puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto("http://example.com", { waitUntil: "networkidle2" });

  await page.setRequestInterception(true);
  page.on("request", request => {
    console.log("Interception:", Object.keys(request.headers()).join());
    request.continue();
  });
  page._client.on("Network.responseReceived", data => {
    console.log(
      "ResponseReceived:",
      Object.keys(data.response.requestHeaders).join()
    );
  });
  await page.evaluate(() => {
    document.cookie = "foo=bar";
  });
  await page.evaluate(() => {
    var myHeaders = new Headers();
    myHeaders.append("X-Custom-Header", "Hello");

    var myInit = {
      method: "GET",
      headers: myHeaders
    };

    var myRequest = new Request("http://example.com/?Asdf", myInit);
    fetch(myRequest);
  });

  await page.waitFor(1000);

  browser.close();
})();

[aslushnikov]

@mattzeunert DevTools Protocol network inspection is located quite high in the network stack. This architecture doesn't let us collect all the headers that are added to the requests. So the ones we report in Network.requestWillBeSent and Network.requestIntercepted are not complete; this will stay like this for the foreseeable future.

There are a few ways to get real request headers:

• the crude one is to use proxy
• the more elegant one is to rely on Network.responseReceived DevTools protocol event. The actual headers are reported there as requestHeaders field in the Network.Response.

Let me know if this somehow helps.

[mattzeunert]

Thanks @aslushnikov! The idea was to do the request myself in Node rather than letting Chrome do it. So I can't wait for the responseReceived event.

I'm going to use a proxy for now.

[aslushnikov]

@mattzeunert yeah I see; hope it'll work out nicely.

[alkorsan]

@aslushnikov using the new chrome v73 Network.responseReceived does not contain requestHeaders, only the first request have it. using --disable-site-isolation-trials --disable-features=IsolateOrigins,site-per-process --disable-web-security have no effect too.
is there anything I can do to get the full headers of the requests with the new chrome versions? thank you