Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(plugin-stealth): Add new user-agent-language-platform plugin #104

Merged
merged 7 commits into from Jan 6, 2020
Merged

Conversation

Niek
Copy link
Collaborator

@Niek Niek commented Dec 12, 2019

This plugin basically replaces the following evasions:
https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth/evasions/accept-language
https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth/evasions/navigator.languages
At the same time, it fixes #91

Instead of changing request headers and the navigator.languages (navigator.language was missing by the way) properties as in the evasions above, we use call setUserAgentOverride in the CDP Network domain directly which is a better way of achieving this.

In addition, this evasion allows setting the navigator.platform and navigator.vendor properties, which makes it a lot less easy to detect puppeteer.

@berstend
Copy link
Owner

berstend commented Dec 14, 2019

@Niek Interesting, thanks a lot for this PR!

I wasn't aware that you could modify the Accept-Language request header using Network.setUserAgentOverride.

Would you mind adding a test file that does before/after tests with this plugin?
You could use e.g. this file as the basis: https://github.com/berstend/puppeteer-extra/blob/master/packages/puppeteer-extra-plugin-stealth/evasions/accept-language/index.test.js

I would do it myself but there's no good way in Github (that I'm aware of) to push commits to this PR (without a new PR and you losing the "Contributor" status). :)

Is there a specific reason to add navigator.vendor here as well, as opposed to making a specific plugin for it? I personally lean towards making the evasions as modular as possible and splitting them up if it makes sense.

The rest (userAgent, acceptLanguage, platform) is set by a single Network.setUserAgentOverride call, so it make sense to group it into a single plugin.

Thanks again!

@berstend
Copy link
Owner

berstend commented Dec 14, 2019

#91 is still something I need to look into (so end users can use the adblocker plugin while still being able to do their own request interception), but if this PR fixes the accept-language evasion then this would make fixing #91 less critical in my opinion.

@berstend
Copy link
Owner

Once you add a test file you can specifically run it like so (from the root of the puppeteer-extra-plugin-stealth folder):

yarn ava -v ./evasions/webgl.vendor/index.test.js

@Niek
Copy link
Collaborator Author

Niek commented Dec 14, 2019

@berstend Thank you for the amazing repo! I will add the tests ASAP to the PR.

I have removed the navigator.vendor part, I agree that would probably be better in a separate plugin.

@Niek
Copy link
Collaborator Author

Niek commented Dec 16, 2019

I added the tests, let me know if you need other changes @berstend

@Niek Niek requested a review from berstend December 16, 2019 11:33
@Niek Niek mentioned this pull request Dec 16, 2019
@berstend
Copy link
Owner

@Niek awesome, thanks a lot! Will look at the PRs asap. :)

@berstend berstend changed the title Added new user-agent-language-platform plugin feat(plugin-stealth): Add new user-agent-language-platform plugin Jan 6, 2020
@berstend berstend merged commit 3ae8066 into berstend:master Jan 6, 2020
@berstend
Copy link
Owner

berstend commented Jan 6, 2020

@Niek thanks again and apologies for the delay (I'd like to blame the holidays) 😄

I've removed the user-agent, accept-language evasions which are made obsolete by this one and renamed it user-agent-override (to keep it more concise).

Will merge in some other fixes & PRs and do a release afterwards. :)

Thanks! 👍

@berstend berstend removed their request for review January 6, 2020 11:44
@berstend
Copy link
Owner

berstend commented Jan 6, 2020

Released in puppeteer-extra-plugin-stealth@2.4.7. :) Thanks again!

@Niek
Copy link
Collaborator Author

Niek commented Jan 6, 2020

Awesome, thanks for the great work!

@pastuh
Copy link

pastuh commented Jan 8, 2020

What if i want to randomize User Agent?
Can i somehow turn off user-agent-override but still use StealthPlugin ?

Use separate plugins or there exists skip function?
Edit: Used separate override.. looks like works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Request interception with multiple plugins enabled
3 participants