[puppeteer-extra-plugin-block-resources] Error: Request is already handled! when using along side puppeteer-extra-plugin-adblocker

[andrewgadziksonos]

const puppeteer = require("puppeteer-extra");
const pluginStealth = require("puppeteer-extra-plugin-stealth");
const pluginUa = require("puppeteer-extra-plugin-anonymize-ua");
const pluginAdBlocker = require("puppeteer-extra-plugin-adblocker");
const pluginBlockResources = require("puppeteer-extra-plugin-block-resources");
const pluginClickWait = require("puppeteer-extra-plugin-click-and-wait");
const url = "https://www.bestbuy.com";

puppeteer.use(pluginStealth());
puppeteer.use(pluginUa());
puppeteer.use(pluginBlockResources({
  blockedTypes: new Set(['image', 'stylesheet'])
}));
puppeteer.use(pluginAdBlocker({
  blockTrackers: true
}));
puppeteer.use(pluginClickWait());

(async function() {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  await page.goto(url);

  await browser.close();
})();

Running the code above generates the following errors (one error per request!)

(node:7860) UnhandledPromiseRejectionWarning: Error: Request is already handled!
    at assert (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\helper.js:283:11)
    at Request.continue (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\NetworkManager.js:427:5)
    at Request.<anonymous> (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\helper.js:112:23)
    at Plugin.onRequest (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer-extra-plugin-block-resources\index.js:101:60)
    at Page.emit (events.js:208:15)
    at NetworkManager.<anonymous> (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\Page.js:111:68)
    at NetworkManager.emit (events.js:208:15)
    at NetworkManager._onRequest (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\NetworkManager.js:234:10)
    at NetworkManager._onRequestPaused (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\NetworkManager.js:210:12)
    at CDPSession.emit (events.js:203:13)
  -- ASYNC --
    at Request.<anonymous> (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\helper.js:111:15)
    at Plugin.onRequest (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer-extra-plugin-block-resources\index.js:101:60)
    at Page.emit (events.js:208:15)
    at NetworkManager.<anonymous> (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\Page.js:111:68)
    at NetworkManager.emit (events.js:208:15)
    at NetworkManager._onRequest (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\NetworkManager.js:234:10)
    at NetworkManager._onRequestPaused (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\NetworkManager.js:210:12)
    at CDPSession.emit (events.js:203:13)
    at CDPSession._onMessage (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\Connection.js:200:12)
    at Connection._onMessage (C:\Users\agadzik\Documents\git-repos\amz-shop-bot\node_modules\puppeteer\lib\Connection.js:112:17)
(node:7860) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 322)

[berstend]

Hmm, damn - I actually went to great length to avoid that :-/

You have two options until I fix that (I'm busy the next days so the fix won't be as quick as usual):

const PluginStealth = require("puppeteer-extra-plugin-stealth");
const pluginStealth = PluginStealth()

pluginStealth.enabledEvasions.delete('accept-language')

puppeteer.use(pluginStealth)

Or use an older version (yarn add puppeteer-extra-plugin-stealth@2.4.0)

[berstend]

Closing in favor of #91

[berstend]

I just released a hotfix to disable accept-language by default until this is fixed. New version: puppeteer-extra-plugin-stealth@2.4.5

[andrewgadziksonos]

thank you for the quick response and fix!

[kaiserdj]

If i try to use:

const puppeteer = require("puppeteer-extra");
puppeteer.use(require("puppeteer-extra-plugin-stealth")());
puppeteer.use(require("puppeteer-extra-plugin-block-resources")({
    "blockedTypes": new Set(["stylesheet", "font", "texttrack", "other"])
}));
puppeteer.use(require("puppeteer-extra-plugin-adblocker")({"blockTrackers": false}));

The same thing happens to me, I use the latest version

[darkguy2008]

Same here as @kaiserdj says, using:

    "puppeteer-extra-plugin-adblocker": "^2.11.3",
    "puppeteer-extra-plugin-block-resources": "^2.2.4",
    "puppeteer-extra-plugin-stealth": "^2.4.9",

and

puppeteer.use(BlockResourcesPlugin({ blockedTypes: new Set(this.OPTIONS.skipResources) }));
const PluginStealth = require("puppeteer-extra-plugin-stealth");
const pluginStealth = PluginStealth();
pluginStealth.enabledEvasions.delete('accept-language');
puppeteer.use(pluginStealth);
puppeteer.use(AdblockerPlugin({ blockTrackers: true })); // It happens once I add AdBlockerPlugin :(

[pyprism]

Same here as @kaiserdj says, using:

    "puppeteer-extra-plugin-adblocker": "^2.11.3",
    "puppeteer-extra-plugin-block-resources": "^2.2.4",
    "puppeteer-extra-plugin-stealth": "^2.4.9",

and

puppeteer.use(BlockResourcesPlugin({ blockedTypes: new Set(this.OPTIONS.skipResources) }));
const PluginStealth = require("puppeteer-extra-plugin-stealth");
const pluginStealth = PluginStealth();
pluginStealth.enabledEvasions.delete('accept-language');
puppeteer.use(pluginStealth);
puppeteer.use(AdblockerPlugin({ blockTrackers: true })); // It happens once I add AdBlockerPlugin :(

I am also facing this issue, if I disabled AdblockerPlugin, then the problem disappeared .

[NawarA]

I can confirm this is still an open issue. I'm using the latest Puppeteer (v5) and the latest versions of the puppeteer-extra, and all of the plugins. If I disable adblocker, the issue goes away. Enabling adblocker creates the issue.

Is anyone working on a patch for this?

This crashes the browser, so it should be considered a critical bug

[brunogaspar]

Is anyone working on a patch for this?

As far as i know, no one is working on it, so you can attempt to fix it if you want :)

[aconchillo]

I consistently hit the same issue. The problem in my case was that I was calling puppeteer.use(AdblockerPlugin()) every time before calling puppeteer.launch.

The plugin only needs to be registered once. At least this solved my problem.

[berstend]

@aconchillo interesting. Keep in mind that the imports are being cached, if you need fresh puppeteer-extra instances use addExtra instead :-)

[aconchillo]

@berstend thanks! is it safe to use the same puppeteer-extra instance with multiple browsers?

That is.

import puppeteer from 'puppeteer-extra';

// only once
puppeteer.use(StealthPlugin());

// can be called multiple times.
browser = await puppeteer.launch(...);

[berstend]

@aconchillo Yep, using multiple browsers with the same instance should do fine.

I've had this use-case in mind when I implemented the plugin framework:
https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin#afterlaunchbrowser-opts

Note: Don't assume that there will only be a single browser instance during the lifecycle of a plugin. It's possible that pupeeteer.launch will be called multiple times and more than one browser created. In order to make the plugins as stateless as possible don't store a reference to the browser instance in the plugin but rather consider alternatives.

[aconchillo]

@aconchillo Yep, using multiple browsers with the same instance should do fine.

I've had this use-case in mind when I implemented the plugin framework:
https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin#afterlaunchbrowser-opts

Nice. Great to hear! This project is great btw! 🙌

[berstend]

@aconchillo Thanks! Soon it will get even better (#303) 😄

[belvederef]

I can confirm that the issue still happens and is very confusing. Using both puppeteer-extra-plugin-block-resources and puppeteer-extra-plugin-adblocker triggers an unhandledRejection, that crashes the app the does not get caught even when all the logic is enclosed in a try...catch block.

I think at least a disclaimer should be put on both plugins doc pages advising to not use them together.

[Hardik500]

The issue is still present, have we found any workaround on how to handle requests.