2captcha stopped working on Google Sites

[dantgal]

It was working like a charm until yesterday. Google reCaptcha is solved but it not redirects to show search result page after solving captcha...

My code is:

// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality
const puppeteer = require('puppeteer-extra')

// add recaptcha plugin and provide it your 2captcha token (= their apiKey)
// 2captcha is the builtin solution provider but others would work as well.
// Please note: You need to add funds to your 2captcha account for this to work
const RecaptchaPlugin = require('puppeteer-extra-plugin-recaptcha')
puppeteer.use(
  RecaptchaPlugin({
    provider: {
      id: '2captcha',
      token: 'd4e...' // REPLACE THIS WITH YOUR OWN 2CAPTCHA API KEY ⚡
    },
    visualFeedback: true // colorize reCAPTCHAs (violet = detected, green = solved)
  })
)

// puppeteer usage as normal
puppeteer.launch({ headless: true }).then(async browser => {
  const page = await browser.newPage()
  //await page.goto('https://www.google.com/recaptcha/api2/demo')
  await page.goto('https://www.google.com/search?q=klima&oq=klima&aqs=chrome..69i57j0l5j69i60l2.735j0j7&sourceid=chrome&ie=UTF-8')

  // That's it, a single line of code to solve reCAPTCHAs 🎉
  await page.solveRecaptchas()

  await Promise.all([
    page.waitForNavigation()
  ])
  let bodyHTML = await page.evaluate(() => document.body.innerHTML);
  console.log(bodyHTML)
  await page.screenshot({ path: 'response.png', fullPage: true })
  await browser.close()
})

and here's the debug:

seo@serv:~/google$ DEBUG=puppeteer-extra,puppeteer-extra-plugin:* node test.js

  puppeteer-extra-plugin:base:recaptcha Initialized. +0ms
  puppeteer-extra-plugin:recaptcha Initialized {
  visualFeedback: true,
  throwOnError: false,
  provider: { id: '2captcha', token: 'd4e...' }
} +0ms
  puppeteer-extra plugin registered recaptcha +0ms
  puppeteer-extra no dependencies are missing +1ms
  puppeteer-extra orderPlugins:before [ 'recaptcha' ] +0ms
  puppeteer-extra orderPlugins:after [ 'recaptcha' ] +0ms
  puppeteer-extra-plugin:recaptcha onPageCreated about:blank +0ms
  puppeteer-extra-plugin:recaptcha solveRecaptchas +0ms
  puppeteer-extra-plugin:recaptcha findRecaptchas +0ms
  puppeteer-extra-plugin:recaptcha hasRecaptchaScriptTag true +0ms
  puppeteer-extra-plugin:recaptcha waitForRecaptchaClient - start 2020-05-23T22:04:15.081Z +0ms
  puppeteer-extra-plugin:recaptcha waitForRecaptchaClient - end 2020-05-23T22:04:15.091Z +0ms
  puppeteer-extra-plugin:recaptcha _generateContentScript findRecaptchas undefined +0ms
  puppeteer-extra-plugin:recaptcha findRecaptchas {
  captchas: [
    {
      sitekey: '6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b',
      callback: 'submitCallback',
      id: 'oda83noeflau',
      widgetId: 0,
      display: [Object],
      url: 'https://www.google.com/sorry/index?continue=https://www.google.com/search%3Fq%3Dklima%26oq%3Dklima%26aqs%3Dchrome..69i57j0l5j69i60l2.735j0j7%26sourceid%3Dchrome%26ie%3DUTF-8&q=EgS8pdi4GN67pvYFIhkA8aeDS9yQqQWq3BMCa6xUSD_F653Fxcx7MgFy',
      hasResponseElement: true
    }
  ],
  error: null
} +0ms
  puppeteer-extra-plugin:recaptcha getRecaptchaSolutions +0ms
  puppeteer-extra-plugin:recaptcha:2captcha Requesting solution.. {
  provider: '2captcha',
  id: 'oda83noeflau',
  requestAt: 2020-05-23T22:04:15.100Z
} +0ms
  puppeteer-extra-plugin:recaptcha:2captcha Got response {
  err: null,
  result: {
    id: '63956086379',
    text: '03AGdBq26iX0JWbJGTu1Bbl98SH2ZQQxAsLELiNrf9Iz14ORUZPvdjRgNPuF2dgpERJb1ylQqQyw4dSDs8K-UruIlaeE8K9c064cCHYOeuwvggqV91bzYkZDbX39NUToTr7UDgO0LOTts803ELdwNxp5FRZutEqpwK4aTjzX9u8ROXJ1-v4ozbJofb38-zVnhm1eftXYpLxllJpXonswmha5GJRPIwPjKv4RZCdz6xTQRN1wtOPX4emUPBj2qadsKpTg633JV46sh2c66vT7Syb4BSFajNFfR_GbbFdPlnDcd5_E4gg3kV58c'
  },
  invalid: [Function: bound ]
} +2s
  puppeteer-extra-plugin:recaptcha getRecaptchaSolutions {
  solutions: [
    {
      provider: '2captcha',
      id: 'oda83noeflau',
      requestAt: 2020-05-23T22:04:15.100Z,
      providerCaptchaId: '63956086379',
      text: '03AGdBq26iX0JWbJGTu1Bbl98SH2ZQQxAsLELiNrf9Iz14ORUZPvdjRgNPuF2dgpERJb1ylQqQyw4dSDs8K-UruIlaeE8K9c064cCHYOeuwvggqV91bzYkZDbX39NUToTr7UDgO0LOTts803ELdwNxp5FRZutEqpwK4aTjzX9u8ROXJ1-v4ozbJofb38-zVnhm1eftXYpLxllJpXonswmha5GJRPIwPjKv4RZCdz6xTQRN1wtOPX4emUPBj2qadsKpTg633JV46sh2c66vT7Syb4BSFajNFfR_GbbFdPlnDcd5_E4gg3kV58c',
      responseAt: 2020-05-23T22:04:17.370Z,
      hasSolution: true,
      duration: 2.27
    }
  ],
  error: undefined
} +0ms
  puppeteer-extra-plugin:recaptcha enterRecaptchaSolutions +0ms
  puppeteer-extra-plugin:recaptcha _generateContentScript enterRecaptchaSolutions {
  solutions: [
    {
      provider: '2captcha',
      id: 'oda83noeflau',
      requestAt: 2020-05-23T22:04:15.100Z,
      providerCaptchaId: '63956086379',
      text: '03AGdBq26iX0JWbJGTu1Bbl98SH2ZQQxAsLELiNrf9Iz14ORUZPvdjRgNPuF2dgpERJb1ylQqQyw4dSDs8K-UruIlaeE8K9c064cCHYOeuwvggqV91bzYkZDbX39NUToTr7UDgO0LOTts803ELdwNxp5FRZutEqpwK4aTjzX9u8ROXJ1-v4ozbJofb38-zVnhm1eftXYpLxllJpXonswmha5GJRPIwPjKv4RZCdz6xTQRN1wtOPX4emUPBj2qadsKpTg633JV46sh2c66vT7Syb4BSFajNFfR_GbbFdPlnDcd5_E4gg3kV58c',
      responseAt: 2020-05-23T22:04:17.370Z,
      hasSolution: true,
      duration: 2.27
    }
  ]
} +0ms
  puppeteer-extra-plugin:recaptcha enterRecaptchaSolutions {
  solved: [
    {
      id: 'oda83noeflau',
      responseElement: true,
      responseCallback: true,
      isSolved: true,
      solvedAt: {}
    }
  ],
  error: undefined
} +0ms
  puppeteer-extra-plugin:recaptcha solveRecaptchas {
  captchas: [
    {
      sitekey: '6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b',
      callback: 'submitCallback',
      id: 'oda83noeflau',
      widgetId: 0,
      display: [Object],
      url: 'https://www.google.com/sorry/index?continue=https://www.google.com/search%3Fq%3Dklima%26oq%3Dklima%26aqs%3Dchrome..69i57j0l5j69i60l2.735j0j7%26sourceid%3Dchrome%26ie%3DUTF-8&q=EgS8pdi4GN67pvYFIhkA8aeDS9yQqQWq3BMCa6xUSD_F653Fxcx7MgFy',
      hasResponseElement: true
    }
  ],
  solutions: [
    {
      provider: '2captcha',
      id: 'oda83noeflau',
      requestAt: 2020-05-23T22:04:15.100Z,
      providerCaptchaId: '63956086379',
      text: '03AGdBq26iX0JWbJGTu1Bbl98SH2ZQQxAsLELiNrf9Iz14ORUZPvdjRgNPuF2dgpERJb1ylQqQyw4dSDs8K-UruIlaeE8K9c064cCHYOeuwvggqV91bzYkZDbX39NUToTr7UDgO0LOTts803ELdwNxp5FRZutEqpwK4aTjzX9u8ROXJ1-v4ozbJofb38-zVnhm1eftXYpLxllJpXonswmha5GJRPIwPjKv4RZCdz6xTQRN1wtOPX4emUPBj2qadsKpTg633JV46sh2c66vT7Syb4BSFajNFfR_GbbFdPlnDcd5_E4gg3kV58c',
      responseAt: 2020-05-23T22:04:17.370Z,
      hasSolution: true,
      duration: 2.27
    }
  ],
  solved: [
    {
      id: 'oda83noeflau',
      responseElement: true,
      responseCallback: true,
      isSolved: true,
      solvedAt: {}
    }
  ],
  error: undefined
} +0ms

<div style="max-width:400px;">
<hr noshade="" size="1" style="color:#ccc; background-color:#ccc;"><br>
<form id="captcha-form" action="index" method="post">
<script type="text/javascript" async="" src="https://www.gstatic.com/recaptcha/releases/BT5UwN2jyUJCo7TdbwTYi_58/recaptcha__en.js"></script><script src="https://www.google.com/recaptcha/api.js" async="" defer=""></script>
<script>var submitCallback = function(response) {document.getElementById('captcha-form').submit();};</script>
<div id="recaptcha" class="g-recaptcha" data-sitekey="6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b" data-callback="submitCallback" data-s="cojAJSZCdLw81Mw5102PzG5LCy2bEgY-Qgb0-Oa8bdMLsAjcUB7GHfYlSTUsLPcez0e4N8BYqFCc-jZ-XabU9E0GKLwuWbMScAFsxh0IqK2nuqHNLT3f4jCHZU113pPZQt8Wc56k250zig5yoYxpMvWQ85CIvEfWfG3BrT60dccP788N2aLdbn12pmWzOsKuLt893rL8KxnpaMHs4ZsiPkUJdCsAy6AzaRAinNs4eV23RDtrGHNi-u0"><div style="width: 304px; height: 78px;"><div><iframe src="https://www.google.com/recaptcha/api2/anchor?ar=1&amp;k=6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b&amp;co=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbTo0NDM.&amp;hl=en&amp;v=BT5UwN2jyUJCo7TdbwTYi_58&amp;size=normal&amp;s=cojAJSZCdLw81Mw5102PzG5LCy2bEgY-Qgb0-Oa8bdMLsAjcUB7GHfYlSTUsLPcez0e4N8BYqFCc-jZ-XabU9E0GKLwuWbMScAFsxh0IqK2nuqHNLT3f4jCHZU113pPZQt8Wc56k250zig5yoYxpMvWQ85CIvEfWfG3BrT60dccP788N2aLdbn12pmWzOsKuLt893rL8KxnpaMHs4ZsiPkUJdCsAy6AzaRAinNs4eV23RDtrGHNi-u0&amp;cb=r91kon3sihd2" width="304" height="78" role="presentation" name="a-hui2ii7cbhhz" frameborder="0" scrolling="no" sandbox="allow-forms allow-popups allow-same-origin allow-scripts allow-top-navigation allow-modals allow-popups-to-escape-sandbox"></iframe></div><textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response" style="width: 250px; height: 40px; border: 1px solid rgb(193, 193, 193); margin: 10px 25px; padding: 0px; resize: none; display: none;"></textarea></div><iframe style="display: none;"></iframe></div>
<input type="hidden" name="q" value="EgS8pdi4GN67pvYFIhkA8aeDS9yQqQWq3BMCa6xUSD_F653Fxcx7MgFy"><input type="hidden" name="continue" value="https://www.google.com/search?q=klima&amp;oq=klima&amp;aqs=chrome..69i57j0l5j69i60l2.735j0j7&amp;sourceid=chrome&amp;ie=UTF-8">
</form>
<hr noshade="" size="1" style="color:#ccc; background-color:#ccc;">

<div style="font-size:13px;">
<b>About this page</b><br><br>

Our systems have detected unusual traffic from your computer network.  This page checks to see if it's really you sending the requests, and not a robot.  <a href="#" onclick="document.getElementById('infoDiv').style.display='block';">Why did this happen?</a><br><br>

<div id="infoDiv" style="display:none; background-color:#eee; padding:10px; margin:0 0 15px 0; line-height:1.4em;">
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the <a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop.  In the meantime, solving the above CAPTCHA will let you continue to use our services.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  <a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.
</div>

IP address: x.x.x.x<br>Time: 2020-05-23T22:04:17Z<br>URL: https://www.google.com/search?q=klima&amp;oq=klima&amp;aqs=chrome..69i57j0l5j69i60l2.735j0j7&amp;sourceid=chrome&amp;ie=UTF-8<br>
</div>
</div>

Tried also with iframe solution, but at the debug it hasn't found any captcha.
How to make it work again?

[marcio199226]

In my case if recaptcha is embedded within a page it works, but this is no longer working (frame recaptcha):

      for (const frame of page.mainFrame().childFrames()) {
        await frame.solveRecaptchas()
      }

That code does not find any captcha anymore if used in iframe.

[dantgal]

yep... I have same, the strangest thing is that when iframe the page.solveRecaptchas() finds the captcha sends it to solve, get results and page not reloads - maybe some problem with serving the solution. I have tried to reload frames - there was also no redirect to the search results page after solving captcha.
Any ideas?

[dantgal]

2captcha staff solved problem: https://2captcha.com/blog/google-search-recaptcha - plugin needs to be updated to send also data-s variable

[berstend]

Quick note: Fixing this is high on our list (Puppeteer 5 support was higher and is fixed now: #228,#243). :-)

[andrew-healey]

Have we considered, long-term, using a generalized CAPTCHA solving library as a backend for puppeteer-extra-plugin-recaptcha so that other providers are supported (e.g. AntiCaptcha, CAPTCHAS.io AKA #237)? I think it would not only add more providers (and potentially more types of CAPTCHAs, like GeeTest, hCAPTCHA, etc.), it would also be good to reduce the complexity of the codebase. Currently, the options on npm do not have wide support, but there are a few packages which support a) adding providers to the package or b) using your own external provider class. From a bit of research, I think the best is captcha-solver.

[andrew-healey]

Also, about this specific issue, do we know how we plan on stopping data-s from being consumed by reCAPTCHA? The article posted by 2captcha says that

2. You should NOT run these JS which loads the captcha on google.com/sorry/index. If you load the captcha on the webpage and then send us this captcha for solving, a token we provide with a solution then will NOT work. That is because of this new data-s variable. A captcha could be loaded only once with given data-s.

I'm currently working on a PR to solve this (as it is a higher priority than -plugin-stealth) and think that our current method of detecting reCAPTCHAs does not work on Google. As our current method traverses the ___grecaptcha_cfg object, it requires the reCAPTCHA script to have run already. But since 2captcha says that you need to block Google reCAPTCHA from running, it seems to me like fixing this issue might require changing the way we use reCAPTCHA. I'm no expert on reCAPTCHA by any meaning of the word, so I would appreciate being corrected.

[andrew-healey]

Good news: it seems that blocking any request to https://www.google.com/recaptcha/api/fallback and https://www.google.com/recaptcha/api2/anchor will allow ___grecaptcha_cfg to be set without ever using up the data-s token. I'm fairly sure that this URL blocking setup can be used across all websites as long as puppeteer-extra-plugin-recaptcha is active.

[jimspillane]

There were some updates in the forum google search 18th of May, 2020 indicating that data-s is no longer needed.

[berstend]

Have we considered, long-term, using a generalized CAPTCHA solving library as a backend for puppeteer-extra-plugin-recaptcha

Yup, back in the day my intention was to use transitive-bullshit's captcha-solver lib as an abstraction (I appreciate his work, he was actually the one suggesting a more modular approach of this project and is thanked in our Readme). :-)

Unfortunately I ran into issues with captcha-solver and the quickest solution was to be opinionated, remember the "Don't Write Code You Don't Need" mantra and ship our own.

As the current recaptcha issues show it could make sense for us to keep this functionality in-house, as it's a core dependency to our plugin (setting up our own sturdy & typesafe captcha-solver library is not a lot of work). So far there just hasn't been a real reason to do it, as 2captcha performed well and was among the cheapest options. :-)

[berstend]

Good news: it seems that blocking any request to https://www.google.com/recaptcha/api/fallback and https://www.google.com/recaptcha/api2/anchor will allow ___grecaptcha_cfg to be set without ever using up the data-s token. I'm fairly sure that this URL blocking setup can be used across all websites as long as puppeteer-extra-plugin-recaptcha is active.

Nice, any sort of generic solution that doesn't affect non-special recaptcha implementations would be great. We can quite easily block those requests in the plugin if that yields the desired effect.

There were some updates in the forum google search 18th of May, 2020 indicating that data-s is no longer needed.

Am I reading this correctly that all these workarounds require a proxy being used and supplied to the captcha solution provider? If possible I'd like to avoid using that as the plugin will lose some of it "it just works" magic if the user has to use a proxy for captchas being solved.

[andrew-healey]

I'm with you about using an in-house library. However, I think that general support for multiple providers and CAPTCHA types is the best solution in the long run. I'm not suggesting to add it in the next few weeks or months (playwright integration, general fixes and CI are much more important than adding solvers), but on-demand/on-request support for different CAPTCHA solvers and types seems to me like the way to go.
One question I do have is how to test the data-s functionality. I was thinking of using Tor and changing Tor nodes until we find one that Google gives a CAPTCHA to, then to run the solver on that. However, that solution seems slow and there might be some other, more reliable way to trigger google.com/sorry that I don't know of (in my experience, just copying the URL does not serve a reCAPTCHA).

[GJuniarto]

Are this is still open ? actualy last month i already bypass that 'sorry/index' google update using 2captcha services. All u have to do is
"If you use browser simulation - you MUST block execution of javascript from https://www.gstatic.com/recaptcha/ and https://www.google.com/recaptcha/api.js, or just load only HTML code of the page outside the browser; "; in my case i set enable interception for reject request from the url so i would get site-key without loading the javascript it self and place the key from 2captcha services at the html. Hope this is helping u if this issue still open

[obfuscatus]

It's 2 months and this issue still in progress?

[rationalthinker1]

Are this is still open ? actualy last month i already bypass that 'sorry/index' google update using 2captcha services. All u have to do is
"If you use browser simulation - you MUST block execution of javascript from https://www.gstatic.com/recaptcha/ and https://www.google.com/recaptcha/api.js, or just load only HTML code of the page outside the browser; "; in my case i set enable interception for reject request from the url so i would get site-key without loading the javascript it self and place the key from 2captcha services at the html. Hope this is helping u if this issue still open

How do I get the data-s key and how do I pass it to 2captcha? Can you post a sample code?

[sidikhanrei]

The solution for me is to set args like this

const args = [ '--disable-web-security', '--disable-features=IsolateOrigins,site-per-process' ]

I use this plugins to post and share content from my website (simkada.net) to Facebook and others social media.
This plugins work perfectly after adding args on launch.

Thank you

[berstend]

--disable-web-security don't do this, it's extremely dangerous to do that - the website you're scraping could post on your facebook wall if you're signed in. ;-)

[berstend]

It's 2 months and this issue still in progress?

@obfuscatus You are aware that this is a free, unpaid open-source project right?

Feel free to contribute a fix but don't pollute our issue tracker with this attitude.

No one here contributed any cohesive, substantial fact finding and I currently don't have time to do it myself.

Open question:

• Is a proxy now mandatory to pass to 2captcha for reCAPTCHAs to be solvable on certain sites? Is that on all google services or only specific sites? What is the pattern here to detect this programmatically?

[sidikhanrei]

--disable-web-security don't do this, it's extremely dangerous to do that - the website you're scraping could post on your facebook wall if you're signed in. ;-)

Tested with args just --disable-features=IsolateOrigins,site-per-process and it's work like a charm.
I think the problem is Origins Check on Google Captcha

[berstend]

@sidikhanrei
How does this relate to Google Services apparently now requiring a proxy to be passed to 2captcha for them to solve the captcha?

[sidikhanrei]

I'm not really understand about that.
I just doing auto post to facebook groups,

Watching without --disable-features=IsolateOrigins,site-per-process
2captcha cannot trigger and never run until timeout.

But it's work when using that args.

[berstend]

Good you found a solution to your problem but I think you were fixing something else :-) This issue is about reCAPTCHA on Google sites getting extra scrutiny and requiring a data-s property and proxy to be passed to 2captcha for them to be able to solve it.

[rationalthinker1]

I got it to work. I've upload the code that I used to get it to work.
https://gist.github.com/rationalthinker1/9467169c3efcb204ff879838dbf1ad7e

Let me know if it helped anyone.

[cnscorpions]

please refer to 2captcha solution https://2captcha.com/blog/update-google-recaptcha It seems to work.

[natanavra]

Here's a solution for you guys
Solver lib and example included
https://gist.github.com/natanavra/ac97b859403e13570ac9f1552220e9ab

[cnscorpions]

@natanavra I have given up, Google Recaptcha is truly a nightmare.

[berstend]

Fix is on the way: #366

[berstend]

Support for the reCAPTCHA dialect on Google sites (Search, etc) has been released in puppeteer-extra-plugin-recaptcha@3.2.0.