-
-
Notifications
You must be signed in to change notification settings - Fork 743
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Doesn't play nice with Webpack #93
Comments
Work-around is to import and apply the plugins manually: import puppeteerVanilla from 'puppeteer';
import { addExtra } from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
import AcceptLanguagePlugin from 'puppeteer-extra-plugin-stealth/evasions/accept-language';
import ChromeRuntimePlugin from 'puppeteer-extra-plugin-stealth/evasions/chrome.runtime';
import ConsoleDebugPlugin from 'puppeteer-extra-plugin-stealth/evasions/console.debug';
import IFrameContentWindowPlugin from 'puppeteer-extra-plugin-stealth/evasions/iframe.contentWindow';
import MediaCodecsPlugin from 'puppeteer-extra-plugin-stealth/evasions/media.codecs';
import NavigatorLanguagesPlugin from 'puppeteer-extra-plugin-stealth/evasions/navigator.languages';
import NavigatorPermissionsPlugin from 'puppeteer-extra-plugin-stealth/evasions/navigator.permissions';
import NavigatorPlugins from 'puppeteer-extra-plugin-stealth/evasions/navigator.plugins';
import WebdriverPlugin from 'puppeteer-extra-plugin-stealth/evasions/navigator.webdriver';
import UserAgentPlugin from 'puppeteer-extra-plugin-stealth/evasions/user-agent';
import WebglVendorPlugin from 'puppeteer-extra-plugin-stealth/evasions/webgl.vendor';
import WindowOuterDimensionsPlugin from 'puppeteer-extra-plugin-stealth/evasions/window.outerdimensions';
async () => {
const puppeteer = addExtra(puppeteerVanilla);
const plugins = [
StealthPlugin(),
AcceptLanguagePlugin(),
ChromeRuntimePlugin(),
ConsoleDebugPlugin(),
IFrameContentWindowPlugin(),
MediaCodecsPlugin(),
NavigatorLanguagesPlugin(),
NavigatorPermissionsPlugin(),
NavigatorPlugins(),
WebdriverPlugin(),
UserAgentPlugin(),
WebglVendorPlugin(),
WindowOuterDimensionsPlugin(),
];
const browser = await puppeteer.launch();
for (const plugin of plugins) {
await plugin.onBrowser(browser);
}
const [ page ] = await browser.pages();
for (const plugin of plugins) {
await plugin.onPageCreated(page);
}
// ...
}; |
Hmm, interesting. May I ask why you opted to use Webpack for a NodeJS project? If you could provide a minimal webpack based project with that issue that'd be great, as I then can take a quick look and see how to best fix this. :) |
I use Webpack with Node because it's a simpler way to use Babel, bundle
const path = require('path');
const TerserPlugin = require('terser-webpack-plugin');
const { env } = process;
const isProduction = env['NODE_ENV'] === 'production';
const mode = isProduction ?
'production' :
'development';
console.log({ mode });
module.exports = {
entry: [
'@babel/polyfill',
'./src/index.js',
],
target: 'node',
devtool: isProduction ? false : 'source-map',
mode,
output: {
path: path.join(__dirname, 'build'),
filename: 'index.js'
},
module: {
rules: [
{
test: /\.m?js$/,
exclude: /(node_modules|bower_components)/,
use: {
loader: 'babel-loader',
options: {
babelrc: true,
},
},
},
{
test: /\.js$/,
loader: 'unlazy-loader'
}
],
},
optimization: {
minimize: isProduction,
minimizer: [
new TerserPlugin(),
],
},
resolve: {
alias: {
'pg-native': path.resolve(__dirname, 'aliases/pg-native'),
},
},
}; |
Thanks for providing the config, I'll look into it when I find time.
So it's to use new ES language features? Bundling and minifying shouldn't matter on backend code (hence my question). I personally used Babel with NodeJS back in the day to be able to use ES6 imports but eventually stopped doing that as I noticed issues with that and figured it's not worth it and causes more problems than it does good :) |
Yes I use the latest ES6 features. I also use loaders for other languages (such as Fable) and data (such as JSON, CSS). Bundling everything can also be useful for targets like AWS lambda. |
I think as rule-of-thumb, only using ES6 |
Got it :) Anyway, |
Couple of avenues I'll explore to fix this:
|
What is the purpose of the internal dependency management? Could a simpler design work like this? import puppeteerExtra from 'puppeteer-extra';
import MyPlugin from 'my-plugin';
const puppeteer = pupeteerExtra({
plugins: [
MyPlugin(),
],
});
async () => {
const browser = await puppeteer.launch();
// etc...
}; |
The above already exists (with The idea behind the dependency plugin system was to make it easy to re-use plugins within plugins. E.g. the The code for that is here: https://github.com/berstend/puppeteer-extra/blob/master/packages/puppeteer-extra/src/index.ts#L326-L334 Thinking of it now (I wrote this in the very first version) I think we don't need to handle that within What I currently don't understand: This should still work with Webpack regardless, as they should transform the |
So yeah, it's just webpack being bad at dynamic imports. This fix should work: |
So basically adding this to the webpack
also you will need to install string-replace-loader: https://github.com/Va1/string-replace-loader |
I meant this usage but with internal dependency management removed entirely.
So the plugins would declare the other plugins that they depend on using
|
Would be great to hear if it fixes the webpack issue, then I can add it to the documentation :)
Unfortunately that won't fix the issue with dynamic imports + webpack, we still need to do dynamic |
Why is this? |
It's always good to question assumptions, I'm a big believer in that :) Let's take the One feature is that the user can add or remove evasions on that list, before I'm not aware of a way to accomplish that without dynamic |
Assuming the fix mentioned in #93 (comment) works I'm gonna demote this issue to "nice to have in a future version". :) |
The stealth plugin could require all evasions statically. This would cover most use-cases. Users who need something more custom could manually import the ones that they need. This wouldn't be much code and the full import list could be copied from the stealth plugin then edited. |
I got this working locally but when I used webpack to bundle it and send it over to aws lambda, this line
results to the error below. Adding "kind-of": "^6.0.2" to the project's package.json does not resolve the problem. Error: Cannot find module 'kind-of' |
I had the same issue. unlazy-loader solved this for me... |
@levz0r Interesting. Would you mind providing a full example using |
Hey, sorry for the long delay... Nothing special actually... Just add
To webpack.config.js. That's it. Hope it helps. |
I'm getting a similar problem to this with |
@berstend None of the solutions above worked for me. Do you have any update on the rework ? |
Did you try #93 (comment) ? It is highly likely to work because it is pure ES6 with static imports. |
After many hours, I managed to make it work, albeit only halfway. I had to use @njlr's linked solution in addition to However even though it works nicely on bot.sannysoft.com (it even has one more passing criteria that my browser), it still gets detected by reCaptcha. Possibly something to do with it being in an iframe ? Interestingly enough, removing puppeteer-extra (and all plugins) and simply adding the Thank you for the quick replies, though. |
@Seblor this is out of the scope of this webpack issue but the
It is, as you can see in the code of the |
I'm also getting this error. I'm also using vue webpack with this the same configs. StealthPlugin() is not working for me. I need help TIA |
vue.config
Im using VueJS and ElectronJS for my project, this config of mine is working in dev but once I build the project to prod I receive that error above. Any Idea? thanks in advance! By the way Im using this |
@dannielibor well, depending on your level of dedication you could manually import the stealth evasions as mentioned higher up :-)
A transient ischemic attack (TIA) is a temporary period of symptoms similar to those of a stroke. |
Hi @berstend thanks for your response, but I already tried all that even applying those webpack configs mentioned above.
|
By the way, TIA = thanks in advance :D |
I can confirm that it is working fine in a simple node project but after integrating it to my vue-electron project its not working |
just upgrade clone-deep version |
💯 what @godtoy said. I've fiddled with this just to also get to the point that the |
As @godtoy says, I updated clone-deep to 4.0.1, but I'm getting the same error. I also put the following in webpack.json, but it didn't change. What should I do?
|
@godtoy |
Hi guys, I made it working. Want to share what I got. Hope it works to y'all too. Inside var webpack = require('webpack')
module.exports = {
transpileDependencies: [
'vuetify'
],
pluginOptions: {
electronBuilder: {
externals: [
'puppeteer',
'puppeteer-extra',
'puppeteer-extra-plugin-stealth'
],
nodeModulesPath: ['../../node_modules', './node_modules'],
nodeIntegration: true,
builderOptions: {
asarUnpack: 'node_modules/puppeteer/.local-chromium/**/*',
copyright: 'Copyright © 2021',
nsis: {
oneClick: false,
allowToChangeInstallationDirectory: true
},
publish: [
{
provider: '<your provider>',
repo: '<your repo here>',
owner: '<owner>',
token: '<token here>',
releaseType: 'draft',
publishAutoUpdate: true,
private: true
}
]
}
}
},
configureWebpack: {
externals: {
puppeteer: "require('puppeteer')",
'puppeteer-extra': "require('puppeteer-extra')",
'puppeteer-extra-plugin-stealth': "require('puppeteer-extra-plugin-stealth')"
}
}
} Basically you have to expose these packages and include those inside |
@dannielibor |
anyone got the stealth plugin to work with serverless-bundle? tried most suggestions in this thread, but none seem to work. I've yet to look into |
Hello! If you're using webpack and electron I found a fix for this by making sure to include any merge-deep dependencies as an external and to declare the node_modules path relative to your package.json in your package.json as an 'extraResource'. |
@JasonHoku can you give a example |
I've trimmed it to relevancy for you here: However I now use a script to automate this and have made it much easier to pack and update multiple modules. update and pack pup
Package.json ex
|
…really bundler-compatible Some people had success with webpack in berstend/puppeteer-extra#93 but webpack is pretty complicated
I use (electron-forge+webpack+typescript+puppeteer-extra) got this error:
I finally got it work, here is what I did: add "externals" in webpack.main.config.ts
|
I also use electron-forge+webpack+typescript+puppeteer-extra. This worked, thank you! |
Edit by @berstend:
see here for the workaround: webpack/webpack#4175 (comment) and another one specific to the stealth plugin: #93 (comment)
Original issue:
When bundling I get this error:
And then at run-time:
Of course,
puppeteer-extra-plugin-stealth
is already in thepackage.json
.The text was updated successfully, but these errors were encountered: