New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support response interception #1191

Open
aslushnikov opened this Issue Oct 27, 2017 · 11 comments

Comments

Projects
None yet
7 participants
@aslushnikov
Copy link
Contributor

aslushnikov commented Oct 27, 2017

As of today, we have "request interception" that allows puppeteer to intercept and amend requests before they are sent to the server.

Similarly to this, it would be really handy to have "Response interception" that would allow to intercept and amend responses from server before they are delivered to the browser.

@aslushnikov

This comment has been minimized.

Copy link
Contributor

aslushnikov commented Nov 1, 2017

For the record: @allada has an upstream patch for this: https://chromium-review.googlesource.com/c/chromium/src/+/740401

@gdamjan

This comment has been minimized.

Copy link

gdamjan commented Jan 24, 2018

@aslushnikov

This comment has been minimized.

Copy link
Contributor

aslushnikov commented Mar 14, 2018

@gdamjan it is merged. We're waiting on a network layer servicification to complete in Chromium before relying on the functionality.

@avimar

This comment has been minimized.

Copy link
Contributor

avimar commented Mar 19, 2018

Is there any other way to achieve this before this lands?

e.g. getting the information and disabling a <script> tag before it runs?

@aslushnikov

This comment has been minimized.

Copy link
Contributor

aslushnikov commented Mar 19, 2018

@avimar you can try implementing response interception manually:

  1. enable request interception
  2. on every request, fetch the response manually
  3. fulfill the request with the result, if needed

There are two approaches for step (2):

  • you can fetch on the node-side. This will require more work to support cookies
  • you can "fetch" from the page using page.evaluate. This will give cookies for free, but you'll need to mark this request with some header so that you don't try to intercept it as well.

Let me know if this helps/works for you.

@gdamjan

This comment has been minimized.

Copy link

gdamjan commented Mar 19, 2018

@aslushnikov that approach doesn't work when:

  • the web page sends an XHR request
  • response is html with iframe
  • it inserts the invisible iframe in the DOM
  • that starts a PDF download
@allada

This comment has been minimized.

Copy link
Member

allada commented Mar 23, 2018

@gdamjan, if memory serves correct, downloading is not handled on the renderer level. This means that the webpage should never know about anything being downloaded, since it's handled on the browser level. To deal with downloads, devtools' would need to hand over all requests for every tab, since the download is not dependent on a renderer.

However, you would be able to intercept the request and responses. This should allow you to see the invisible iframe's request go out, but I do not believe you'd receive the HEADERS response followup, since it changes who owns it (but I don't remember all the details of this case).

@gdamjan

This comment has been minimized.

Copy link

gdamjan commented Mar 23, 2018

I've added more info and example page (a simplification of a real page) here :
#1888

@brownbl1

This comment has been minimized.

Copy link

brownbl1 commented Aug 4, 2018

It's been a little while.. is there any update on this? Looking forward to this feature.

@jsoverson

This comment has been minimized.

Copy link

jsoverson commented Sep 13, 2018

Before this lands in puppeteer you can intercept responses via the remote interface and using Network.getResponseBodyForInterception and intercepting at the HeadersReceived stage. It's a little awkward as an API but it works.

const chromeLauncher = require('chrome-launcher');
const CDP = require('chrome-remote-interface');
const btoa = require('btoa');
const atob = require('atob');

function launchChrome(headless = false) {
  return chromeLauncher.launch({
    chromeFlags: [
      '--window-size=412,732',
      // '--user-data-dir=/tmp/foobar',
      '--enable-logging',
      headless ? '--headless' : ''
    ]
  });
}

(async function () {

  const chrome = await launchChrome();
  const protocol = await CDP({ port: chrome.port });

  const { Page, Runtime, Network } = protocol;
  await Promise.all([Page.enable(), Runtime.enable(), Network.enable()]);

  Runtime.consoleAPICalled(({ args, type }) => console[type].apply(console, args.map(a => a.value)));

  await Network.setRequestInterception({ patterns: [{ urlPattern: '*.js*', resourceType: 'Script', interceptionStage: 'HeadersReceived' }] });

  Network.requestIntercepted(async ({ interceptionId, request, responseStatusCode, responseHeaders }) => {
    console.log(`Intercepted ${request.url} {interception id: ${interceptionId}}`);

    const response = await Network.getResponseBodyForInterception({ interceptionId });
    const bodyData = response.base64Encoded ? atob(response.body) : response.body;

    const newBody = bodyData + `\nconsole.log('Intercepted and modified ${request.url}');`;

    const newHeaders = [
      'HTTP/1.1 200 OK',
      'Date: ' + (new Date()).toUTCString(),
      'Connection: closed',
      'Content-Length: ' + newBody.length,
      'Content-Type: text/javascript'
    ];

    Network.continueInterceptedRequest({
      interceptionId,
      rawResponse: btoa(newHeaders.join('\r\n') + '\r\n\r\n' + newBody)
    });
  });

  Page.navigate({ url: 'http://stackoverflow.com' });

  Page.loadEventFired(async () => {

    protocol.close();
    chrome.kill();
  });

})();
@masnampi

This comment has been minimized.

Copy link

masnampi commented Nov 26, 2018

i hope this can be done soon :) i really need this

@aslushnikov aslushnikov added the chromium label Dec 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment