Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anti-CAPTCHA no longer works - Instance blocked #1183

Closed
rustynail1984 opened this issue May 21, 2020 · 45 comments
Closed

Anti-CAPTCHA no longer works - Instance blocked #1183

rustynail1984 opened this issue May 21, 2020 · 45 comments
Labels
bug Something isn't working

Comments

@rustynail1984
Copy link

rustynail1984 commented May 21, 2020

Hi there,

unfortunately the Anti-CAPTCHA suddenly stops working.. CAPTCHA requests are sent to Anti-captcha.com and solved, but no more cookies are saved in config.yml.

I run invidious on a cloud server without docker

I also tried it on another server and via Docker, same problem.

From the Docker log I could see the following error:
Exception: Missing hash key: HTTP::Headers::Key(@name="location")

Does anyone have any idea what I can do?

@artths
Copy link

artths commented May 22, 2020

YouTube has changed something and doesn't accept solved CAPTCHAs anymore. I have same problem.

@rustynail1984
Copy link
Author

rustynail1984 commented May 23, 2020

uh-oh, thats not good.

The only real possibility to bypass reCAPTCHA on Server right now is to install Squid Proxy on your server and use it in your browser. Navigate to YouTube and solve the captcha manually. After you have been successfully redirected to YouTube, open the developer console and copy the "GOOGLE_ABUSE_EXEMPTION" cookie and paste it manually into Config.yml. after that restart invidious and it works again for about 1-2 hours.

I wonder is there a Anti-CAPTCHA Service that response with Cookie instead of a Hash like Anti-Captcha.com it does.

@Perflyst
Copy link
Contributor

Perflyst commented May 23, 2020

Disable your anti-CAPTCHA until this is fixed. Invidious sends a lot of CAPTCHAs which cost you money.

Screenshot_20200523_151233

@artths
Copy link

artths commented May 24, 2020

Sometimes YouTube accepts CAPTCHAs instantly, but more often you need to try like 20 more times. I see no pattern here.

@ghost
Copy link

ghost commented May 25, 2020

yea this is affecting search to in invidio us

@rustynail1984
Copy link
Author

Dont Blame the Dev here... Its Googles shit if they change their system so often...

@artths
Copy link

artths commented May 26, 2020

ceb2529
Does it help?

@rustynail1984
Copy link
Author

@artsemionov nope, doesn't work.

@CuloArdido
Copy link

The only real possibility to bypass reCaptcha on Server right now is to install Squid Proxy on your server and use it in your browser.

How feasible would it be to adapt Buster to run on a server?

@Perflyst Perflyst added the bug Something isn't working label May 26, 2020
@Perflyst Perflyst changed the title Anti Captcha not longer works. No Coockie in config.yml Anti Captcha not longer works - Instance blocked May 26, 2020
@elypter
Copy link

elypter commented May 26, 2020

Unfortunately this was bound to happen sooner or later. i hope there is a workaround but it might be necessary to implement a backup method that Google cannot block. and for that the challenge that the invidious server gets has to be answered by the the invidious server. for that invidious would have to proxy it to one of its users receive back the answer and then reply to YouTube's servers like i suggested in the past.

@ghost
Copy link

ghost commented May 27, 2020

For all that want to still use the service just use a different instance https://instances.invidio.us/

@rustynail1984
Copy link
Author

rustynail1984 commented May 27, 2020

I wonder is there a way in invidious to use Multiple Proxies in sequential mode to bypass CAPTCHA?

@alexture
Copy link

For all that want to still use the service just use a different instance https://instances.invidio.us/

Thanks! Just to be sure: this loses our playlists and subscriptions, right?

@rustynail1984
Copy link
Author

rustynail1984 commented May 27, 2020

Just want to let u know that i also contacted Anti-CAPTCHA about this situation.

maybe they find a solution.

Hi,
just wanted to let you know that we're still solving the issue with Google SERP Recaptcha.
26 May, 17:18

Hi, I'll ask my dev team about this.
23 May, 08:12

Hello,

since 2 days Google ReCAPTCHA v2 cant be longer submitted with your Hashes, Proxyless or with Proxy.
Whether via API in my app or even not via browser extension in Chrome.

URL Example: https://www.google.com/sorry/index?continue=https://www.youtube.com/watch%3Fv%3Dl2n2VG2qiZA&q=EgSjrIHrGJLnoPYFIhkA8aeDS-3cRBGE9Wprixovz_RPoQUHHT3VMgFy

Is there any other way to bypass ReCAPTCHA on YouTube? is there a way in Proxy Mode to get the from YouTube Provided Cookie instead a Hash to submit the CAPTCHA? Cookie Example: GOOGLE_ABUSE_EXEMPTION ID=5636cfbb35d9a4ab:TM=1590179233:C=r:IP=xxx.xxx.xxx.xxx-:S=APGng0uJZBHJHUGNe2X-OibwOUTSD0GXFg
22 May, 22:30

@artths
Copy link

artths commented May 27, 2020

Did you try solving CAPTCHA via proxy? Maybe they started to compare IPs of who solves and who posts the CAPTCHA.

@rustynail1984
Copy link
Author

yes i tried it via proxy, i wrote a NodeJS script with puppeteer and Chrome..

@artths
Copy link

artths commented May 27, 2020

@rustynail1984 I mean there is a NoCaptchaTask at anti-captcha.com that requires setting up a proxy on same server that sending CAPTCHA post requests.

@Perflyst Perflyst pinned this issue May 27, 2020
@rustynail1984
Copy link
Author

@artsemionov yes i tried it via NoCaptchaTask and Proxy. But i cant solve the proxy with the responded hash. it redirect me to https://www.google.com/sorry/index
when i paste the Hash into the textarea field and execute submitCallback(); in browser console.

I hope this is the correct way to post the hash..

@omarroth
Copy link
Contributor

For all that want to still use the service just use a different instance https://instances.invidio.us/

Thanks! Just to be sure: this loses our playlists and subscriptions, right?

You can export/import your data under /data_control. Currently playlists will be lost (tracked in #985).

@rustynail1984 Thanks for sharing your results. I tried something similar in add-proxy and encountered the same issue.

@rustynail1984
Copy link
Author

rustynail1984 commented May 28, 2020

i also tried this in invidious.

"clientKey" => CONFIG.captcha_key,
"task"      => {
  "type"       => "NoCaptchaTask",
  "websiteURL" => location.to_s,
  "websiteKey" => site_key,
  "proxyType" => "http",
  "proxyAddress" => "xxx.xxx.xxx.xxx",
  "proxyPort" => "3128",
  "proxyLogin" => "USERNAME",
  "proxyPassword" => "PASSWORD",
  "userAgent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
},

but unfortunately this doesn't work either. Request is sent to Anti-CAPTCHA i see it in Dashboard.

See the following Errors:

Exception: Missing hash key: "taskId"

Exception: Missing hash key: HTTP::Headers::Key(@name="location")

@CodeSapiens
Copy link

Thanks to everyone involved in getting things working.
Stating the obvious, but keep in mind that Google developers also monitor these blogs to see what people are trying. And since the code is open source, they can also reverse-engineer the new work-arounds.
Not sure what the best approach is, but I think viable ideas need to be communicated and implemented behind closed doors in secured channels, and perhaps a new fork needs to be created for closed source, unfortunately. Otherwise this is just a cat/mouse game where the mouse has nowhere to hide.

@comradekingu comradekingu changed the title Anti Captcha not longer works - Instance blocked Anti-CAPTCHA not longer works - Instance blocked May 28, 2020
@comradekingu comradekingu changed the title Anti-CAPTCHA not longer works - Instance blocked Anti-CAPTCHA no longer works - Instance blocked May 28, 2020
@Perflyst
Copy link
Contributor

Just got an email from anti-captcha

Google added in their widget a new "old" temporary token attribute: "data-s". To 
successfully bypass this restriction, you need to send it to us. You'll also 
need to use worker's cookies from google.com domain which we send to you in this 
case.
Details are available in our knowledge base, use word "serp" to find the record.

Maybe this also applies to youtube.com.

@artths
Copy link

artths commented May 28, 2020

Indeed, there is "data-s" in recaptcha page.

@omarroth
Copy link
Contributor

Should be fixed with 6435c7b.

@ghost
Copy link

ghost commented May 29, 2020

For all that want to still use the service just use a different instance https://instances.invidio.us/

Thanks! Just to be sure: this loses our playlists and subscriptions, right?

yes

@rustynail1984
Copy link
Author

rustynail1984 commented May 29, 2020

Should be fixed with 6435c7b.

Unfortunately no, I still get this error Exception: Missing hash key: HTTP::Headers::Key(@name="location") and no cookies in config.yml

and one time i got this error Exception: Missing hash key: "cookies"

@EsmailELBoBDev2
Copy link
Contributor

EsmailELBoBDev2 commented May 29, 2020

What about we use buster (as he said) ? like type some lines of code that uses buster addon to solve captcha for us using watson speech to text or any service else ?

@rustynail1984
Copy link
Author

I'm doing some test in with NodeJS and I was successful. I was able to bypass the captcha on google.com/sorry.
The problem now is that I see a second captcha directly on youtube. I will now write some code to bypass this as well.

After this i will share my Code.

@CodeSapiens
Copy link

As glad as I am someone is figuring this out, I can't help to think that the moment you post the work-around, Google developers will be on it, developing a counter-measure.
I think folks that understand how the technology works, need to just create a closed source solution that we can buy and use. I would be happy to pay for it, and keep Google from knowing how their trackers are being defeated.

@B0pol
Copy link
Contributor

B0pol commented May 29, 2020

No no no, nobody wants private software here. Invidious works very well. YouTube receive too much requests from a single ip address, and therefore send captcha. That's the reason why some little instances still work: they have less users, and send less requests to YouTube.
You can install Invidious (see https://github.com/omarroth/invidious#installation), it will work excellently.

@rustynail1984
Copy link
Author

As mentioned earlier, I was able to bypass Recaptcha on google.com/sorry.

The second problem, I can't solve the second captcha on youtube.com, it tells me hash key (from anti-captcha) is wrong.
But I noticed, when you are logged in with your YouTube account that this captcha does not appear directly on YouTube.
So here in my script I set my login cookies that I copied from Chrome and vola it works.

Here is my nodejs code: https://pastebin.com/Q7YXGSDH
and here my package.json: https://pastebin.com/BqhXvLBd

@rustynail1984
Copy link
Author

rustynail1984 commented May 29, 2020

Should be fixed with 6435c7b.

Ok seems it is 1/2 solved with this fix. It works now for google.com/sorry but not for the second Captcha on youtube.com video page.

"error": "Missing param name: \"player_response\""

EDIT: Ok the YouTube Login Cookies can also be used in Invidious. That bypass the second Captcha on YouTube.

@captnblood
Copy link

just as a quick idea:
i'm using youtube-dl to get stuffs off youtube. i just had to update again it because it wouldn't work anymore. i have to do that from time to time when youtube modify something.
it got me thinking that if youtube-dl is able to access youtube content,
maybe they figured out this captcha thingy and having a peek at youtube-dl code could provide some inspiration for invidious ?

@elypter
Copy link

elypter commented May 29, 2020

youtube-dl probably fails too eventually if use it too much and you will have to solve a captcha in the browser

@unixfox
Copy link
Member

unixfox commented Jun 3, 2020

I've developed a program that exploit a weakness in Google Recaptcha. I'll try to release the source code after my exams so around the end of June. I need to change some part of the code before publishing to the public and I don't have the time for that right now.

But if you would like to have an early access email me at recaptcha@yewtu.be. Just tell me if you are going to use it on Invidious or not and if it's a public instance the URL of the invidious instance.
I cloned the API of Anti-CAPTCHA so you would just need to change the API URL in src/invidious/helpers/jobs.cr. It's free to use because I don't have to pay anything.

Meanwhile, you may use my instance which has this program implemented: https://yewtu.be (I removed the CAPTCHA on the sign up page so that it's less annoying for you to sign up).

Unfortunately it's no longer working for "Google sorry".

@seniorm0ment
Copy link

just as a quick idea:
i'm using youtube-dl to get stuffs off youtube. i just had to update again it because it wouldn't work anymore. i have to do that from time to time when youtube modify something.

I could be wrong, but I believe this is a caching error. I assume you are referring to the HTTP 403 error? youtube-dl --rm-cache-dir should solve this. I could be wrong though if you are referring to some other error.

@wuniversales
Copy link

Meanwhile, you may use my instance which has this program implemented: https://yewtu.be (I removed the CAPTCHA on the sign up page so that it's less annoying for you to sign up).

"Read timed out " :(

@unixfox
Copy link
Member

unixfox commented Jun 5, 2020

Meanwhile, you may use my instance which has this program implemented: https://yewtu.be (I removed the CAPTCHA on the sign up page so that it's less annoying for you to sign up).

"Read timed out " :(

This a known issue: #1132

@SuperSandro2000
Copy link
Contributor

As glad as I am someone is figuring this out, I can't help to think that the moment you post the work-around, Google developers will be on it, developing a counter-measure.
I think folks that understand how the technology works, need to just create a closed source solution that we can buy and use. I would be happy to pay for it, and keep Google from knowing how their trackers are being defeated.

TopJohnWu played this game for a few years with Magisk and Google. They still need to come up with something and we just break it in a few weeks and they waste more money on it.

@iAmNoVa-UXX
Copy link

Sounds about right to be fair

@wuniversales

This comment has been minimized.

@elypter

This comment has been minimized.

@Perflyst Perflyst unpinned this issue Jun 15, 2020
@Perflyst
Copy link
Contributor

Anti captcha is fixed for now.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests