Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
[crunchyroll] Unable to download #25437
Comments
|
i'm having this error too, here is the complete log with the verbose flag: |
|
Same issue. [debug] System config: [] |
|
No problems with cookie.
|
|
I am trying it with a cookie file, and I am still getting the 403 forbidden page on the login step. Edit - If I remove my login and ONLY use the cookie file I can access the files. It is only if you include the login step do you get the 403 error now. |
|
Actually the trouble is that they put CloudFlare JsChallenge back on the login page, so youtube-dl is considered by Crunchyroll as a bot. |
|
Yay, now we need to find a stupid workaround to bypass this stupid JsChallenge... -_- Thanks again CloudFlare ! |
|
@remitamine Uh this isn't a duplicate. |
|
@remitamine That issue which skid9000 commented in was for a completely different project, https://github.com/skid9000/Crunchyroll-Downloader . An issue there isn't a duplicate for this issue. Or am I missing something? |
|
With regard to the cookie file, I have no luck at all, even if I use a cookie file and don't log in: |
|
I tried with a cookies file and it works for me just fine, |
|
The issue is, before whatever change made by CrunchyRoll, youtube-dl could download a video given an URL to its container webpage, without any need to login, and without manually supplying any cookies file. Now it cannot. There is something about the HTTP request made by youtube-dl that CrunchyRoll does not like, that is different from the one made by the web browser (Firefox in my case). In the case of Firefox, a direct URL to a video will cause www.crunchyroll.com to first respond with a 302 Found, some cookies set, and a redirect to the same URL which then loads normally. In the case of youtube-dl, it instead responds with a 301 Moved Permanently, and when the redirect is followed, it responds with 403 Forbidden. Note that Firefox can show the CrunchyRoll webpage and can also start playback, without any need to login into any session. Therefore, technically a valid login should not be required for youtube-dl to download the data. I have tried installing a Firefox extension that exports the cookies from an anonymous (not logged-in) CrunchyRoll page view into a cookies.txt file, and loading this from youtube-dl, and it does not work - it still responds with 301 and then 403 Forbidden. I have noticed that Firefox uses HTTP/2 for its CrunchyRoll requests, and youtube-dl uses HTTP/1.1 . Could be one difference worth investigating. |
|
It's working with cookies here the details : #25294 |
|
@skid9000 Yes i know, i have this two problems Captcha + Unicode problems with youtube-dl. |
|
That's still a problem, you need a cookie file to download anything. |
|
Crunchyroll add hCaptcha ? Maybe this helps : https://blog.skk.moe/post/bypass-hcaptcha/#Cloudflare-Captcha |
|
I think cloudscraper was used in a pr a long time ago but got refused because it's not unlicensed. (don't quote me on that tho) |
|
Update: the youtube-dl HTTPS request is (rightfully) triggering the Cloudflare anti-bot protection by keying on the SSL use of TLS 1.3 by modern browsers vs. TLS 1.2 or older by youtube-dl. That is, Cloudflare probably thinks that the User-Agent advertised by youtube-dl must be used alongside TLS 1.3 or later, and flags the request as a bot if this does not happen. Evidence: given the following text file (in DOS line-ending format for HTTP protocol specification): GET / HTTP/1.1 The following command triggers the Cloudflare blocking: Note: openssl s_client uses TLS 1.2 by default. TLDR: the CrunchyRoll extractor must find some way to force TLS 1.3 SSL negotiation when requesting the pages. |
|
Is this issue supposed to still be closed? |
|
|
Also, FWIW
As a quick confirmation, if I edit the context.options |= ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_TLSv1_2It makes it past the login page finally, but then gives me the following output: My guess is that, for whatever stupid reason, TLS 1.3 is necessary for the extractor to bypass Cloudflare, but then it completely breaks when accessing the video itself. |
|
(I'm an idiot, much simpler to just |
|
Update: I tried getting it to work by adding two different HTTPS handlers into the URL chain, one supporting only TLS 1.3, and another with the default settings (in that order), and it still fails with the same error messages. |
|
The context settings appear to be retained through different calls. I was able to get it to work (in a very hacky, proof-of-concept kind of way) by forcing TLS v1.3 for To be specific, I put this code into the I would not recommend this at all for production because it would interfere with other handlers, but as a proof-of-concept it demonstrates it working. |
|
Can someone re-open this issue and remove the "duplicate" tag ? @dstftw @remitamine |
|
It appears that the Cloudflare change that denied access to unpatched youtube-dl has been reverted. Please somebody else confirm. |
|
Still 403 here. |
|
When I get out of drinks, I'll tell you if it works. |
|
I repeated the openssl s_client test. It appears that now the HTTPS connection is negotiated at TLS 1.3 level, at least for me, without the need to force the TLS version through the command line. However, I can also force TLS 1.2 with the "-tls1_2" switch, and the request is now accepted again. So I am now confused. @skid9000 Do you have access to the "openssl" program? If you use any flavor of Linux, you probably do. Are you able to repeat the openssl test on your side? |
|
Still 403ing for me with stock youtube-dl from git. |
|
@avillacis What are the contents of your |
The contents were quoted here. |
|
I can't get an HTTP error code with your openssl commands |
|
Ah, thanks, I'm sorry I missed that. Without the |
|
So no news for this issue? |
|
with a pull pending "Detect the [debug] System config: [] |
|
403 with crouncyroll, how can i download from it? can you help me? |
|
@RobertusIT Update python to 3.7+ |
|
Example of it working for me. After trying a few more things, I believe the problem is the attempt to use an account. This works This does not work More minimal examples 403: In short - do not have --username and --password arguments when trying to use crunchyroll. I do not know if using .netrc file is any better but it is probable that it has the exact same problem. You may want to test with cookies after you logged in with a browser. |
and others
[crunchyroll] Downloading login page
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
it seems like this is cause by a Cloud flare Captcha.