Missing header in page.on("response") #4918

rackaam · 2019-09-09T10:08:30Z

Steps to reproduce

Tell us about your environment:

Puppeteer version: 1.14
Platform / OS version: Linux Ubuntu
URLs (if applicable):
Node.js version: 10

What steps will reproduce the problem?

Please include code that reproduces the issue.

Launch server.js

const http = require("http");

const hostname = "127.0.0.1";
const port = 5000;

const server = http.createServer((req, res) => {
  if (req.url === "/one") {
    res.statusCode = 200;
    res.setHeader("Content-Type", "text/html");
    res.setHeader("set-cookie", "_one=1");
    res.end('<html><script src="http://127.0.0.1:5000/two"></script></html>');
  } else if (req.url === "/two") {
    res.statusCode = 200;
    res.setHeader("Content-Type", "application/javascript");
    res.setHeader("set-cookie", "_two=2");
    res.end(`console.log('Hello two');`);
  }
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Launch this puppeteer script

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({
    headless: true
  });
  const page = await browser.newPage();

  page.on("response", response => {
    console.log("New response:");
    console.log(response.url());
    console.log(response.headers());
  });

  await page.goto("http://127.0.0.1:5000/one", {
    waitUntil: "load"
  });

  await browser.close();
})();

What is the expected result?
We should get the set-cookie headers for both requests.

What happens instead?

New response:
http://127.0.0.1:5000/one
{ 'content-type': 'text/html',
  'set-cookie': '_one=1',
  date: 'Mon, 09 Sep 2019 09:59:48 GMT',
  connection: 'keep-alive',
  'content-length': '62' }
New response:
http://127.0.0.1:5000/two
{ date: 'Mon, 09 Sep 2019 09:59:48 GMT',
  connection: 'keep-alive',
  'content-length': '25',
  'content-type': 'application/javascript' }

The second set-cookie header is missing.

We can add this lines to the end of the script to confirm the second set-cookie header was successfully interpreted by the browser (but no visible in the response event).

  const client = await page.target().createCDPSession();
  console.log(await client.send("Network.getAllCookies"));

The text was updated successfully, but these errors were encountered:

kdzwinel · 2019-10-08T10:36:21Z

I've also run into this and can reproduce it in v1.20. It looks like puppeteer is not listening to the Network.responseReceivedExtraInfo event that contains the raw headers (you can do that yourself to work around the issue). DevTools are doing just that:

It also might be a Chromium bug since this cookie is not really blocked, so IMO it should appear in the Network.responseReceived headers.

rackaam · 2019-10-08T16:16:10Z

I think it might be related to https://bugs.chromium.org/p/chromium/issues/detail?id=868407

robertm-97 · 2020-07-24T10:42:29Z

I had this same issue but I managed to find a workaround. The solution is to listen on the Network.responseReceivedExtraInfo event suggested by #4918 (comment) and extract the cookies from there.
I've created a class that collects the cookies set in request and looks something like this

const setCookie = require('set-cookie-parser');

class HttpCookieInterceptor {
    foundCookies = [];

    httpSetCookiesKey = 'set-cookie';

    constructor(client) {
        // Setup listener for handling finished request
        client.on('Network.responseReceivedExtraInfo', (response) => this.captureHttpCookies(response));
    }

    static async create(page) {
        // open sessions to DevTools
        const client = await page.target().createCDPSession();
        await client.send('Network.enable');
        // Setup request interceptor rules, in this case we will intercept all request
        await client.send('Network.setRequestInterception', {
            patterns: [{
                urlPattern: '*',
            }],
        });
        await client.on('Network.requestIntercepted', async (e) => {
            // Let the request continue, we don't need anything from here
            await client.send('Network.continueInterceptedRequest', {
                interceptionId: e.interceptionId,
            });
        });
        return new HttpCookieInterceptor(client);
    }

    captureHttpCookies(request) {
        if (request && request.headers && request.headers[this.httpSetCookiesKey]) {
            const cookieString = request.headers[this.httpSetCookiesKey];
            const httpCookies = setCookie.parse(cookieString);
            this.foundCookies = [...this.foundCookies, ...httpCookies];
        }
    }

    getCookies() {
        return this.foundCookies;
    }
}

It can be used like this:

const browser = puppeteer.launch({
        headless: true,
        devtools: true,
});
const page = await browser.newPage();
const httpCookieCollector = await HttpCookieInterceptor.create(page);
// do some navigation here
const httpCookies = httpCookieCollector.getCookies();

NOTE: This collects all cookies, you will need to filter duplicates after
Hope this helps

tobyhinloopen · 2020-08-02T16:43:55Z

@robertm97's reply combined with https://stackoverflow.com/a/49220156 really helped for me. I actually wanted just all headers; I ended up with this code:

async function getApiCallHeaders(page) {
  return new Promise(async (resolve, reject) => {
    resolved = false;
    try {
      const devtools = await page.target().createCDPSession();
      await devtools.send('Network.enable');
      await devtools.send('Network.setRequestInterception', {
        patterns: [{ urlPattern: '*' }],
      });
      devtools.on('Network.requestIntercepted', async (event) => {
        if (resolved) {
          return; // just stop any other request once we get the headers; I only want the headers
        }
        if (/\/me$/.test(event.request.url)) { // < wait for a request matching `/me$`, extract its headers
          resolved = true;
          resolve(event.request.headers);
          return; // just stop any other request once we get the headers; I only want the headers
        }
        await devtools.send('Network.continueInterceptedRequest', {
          interceptionId: event.interceptionId,
        });
      });
      await page.goto("https://...."); // < go to page that loads an app that triggers an API call
    } catch (error) {
      if (!resolved) { // only throw if we didn't resolve earlier; sometimes errors are thrown after being resolved. I don't care about them.
        resolved = true;
        reject(error);
      }
    }
  });
}

I use this code to extract all headers generated by some SPA so I can cheaply continue to call the API using node-fetch without running a browser. IE I login using a browser (because their login flow is convoluted), wait for an API call to happen, extract all headers, kill the browser and continue using the API using node-fetch with the extracted headers.

And93 · 2020-11-05T22:59:23Z

Maybe someone knows the opposite answer, how to add cookies in response (httpRequest.respond(response)) ? 6578

verhovsky · 2021-03-08T11:10:18Z

I switched to using Firefox through playwright, it returns more headers in request.headers:

import playwright from 'playwright';

(async () => {
    const browser = await playwright['firefox'].launch();
    const page = await browser.newPage();

    page.on('request', req => {
        console.log(req.headers());
    });
    await page.goto("https://example.com/");

    await browser.close();
})();

stale · 2022-06-24T16:49:07Z

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

stale · 2022-07-24T17:01:32Z

We are closing this issue. If the issue still persists in the latest version of Puppeteer, please reopen the issue and update the description. We will try our best to accomodate it!

vali-pavel mentioned this issue Jun 4, 2020

The 'Set-Cookie' header is missing from the response headers list #5377

Closed

ritz078 mentioned this issue Aug 23, 2020

Firefox: page.setRequestInterception throws protocol error. #6359

Closed

ritz078 mentioned this issue Feb 2, 2021

Cookie header not set #5039

Closed

verhovsky mentioned this issue Mar 9, 2021

[BUG] page.on('request', r => r.headers) doesn't return all headers on Chromium and Webkit microsoft/playwright#5761

Closed

stale bot added the unconfirmed label Jun 24, 2022

stale bot closed this as completed Jul 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing header in page.on("response") #4918

Missing header in page.on("response") #4918

rackaam commented Sep 9, 2019

kdzwinel commented Oct 8, 2019 •

edited

rackaam commented Oct 8, 2019

robertm-97 commented Jul 24, 2020 •

edited

tobyhinloopen commented Aug 2, 2020 •

edited

And93 commented Nov 5, 2020

verhovsky commented Mar 8, 2021

stale bot commented Jun 24, 2022

stale bot commented Jul 24, 2022

Missing header in page.on("response") #4918

Missing header in page.on("response") #4918

Comments

rackaam commented Sep 9, 2019

Steps to reproduce

kdzwinel commented Oct 8, 2019 • edited

rackaam commented Oct 8, 2019

robertm-97 commented Jul 24, 2020 • edited

tobyhinloopen commented Aug 2, 2020 • edited

And93 commented Nov 5, 2020

verhovsky commented Mar 8, 2021

stale bot commented Jun 24, 2022

stale bot commented Jul 24, 2022

kdzwinel commented Oct 8, 2019 •

edited

robertm-97 commented Jul 24, 2020 •

edited

tobyhinloopen commented Aug 2, 2020 •

edited