Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crunchyroll Login page throwing 503 #11572

Open
Xonshiz opened this issue Jan 1, 2017 · 108 comments
Open

Crunchyroll Login page throwing 503 #11572

Xonshiz opened this issue Jan 1, 2017 · 108 comments

Comments

@Xonshiz
Copy link

@Xonshiz Xonshiz commented Jan 1, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.12.22. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • [x ] I've verified and I assure that I'm running youtube-dl 2016.12.22

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

Youtube-DL>youtube-dl -i "http://www.crunchyroll.com/fairy-musketeers/episode-1-akazukin-finally-comes-510412" -u MyUserName -p MyPassword --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-i', 'http://www.crunchyroll.com/fairy-musketeers/episode-1-akazukin-finally-comes-510412', '-u', 'PRIVATE', '-p', 'PRIVATE', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2016.12.31
[debug] Python version 3.4.4 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-62728-g6956b04, rtmpdump 2.5
[debug] Proxy map: {}
[Crunchyroll] Downloading login page
ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest versi
on; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpa0c7afb5\build\youtube_dl\extractor\common.py", line 406, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpa0c7afb5\build\youtube_dl\YoutubeDL.py", line 2001, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

...

Description of your issue, suggested solution and other information

If you provide login credentials for Crunchyroll, then the script shows a HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()) error. But, if you remove the login credentials, it'll download the video/playlist just fine. Above, I have provided a log with the login credentials. I can open the login page just fine on all the browsers and I'm not behind any firewall. I have tried this with 2 different systems, in different location and I had the same result.

I checked the login page and it was just cloudflare browser check. There was no captcha to solve or anything, as mentioned in this issue.

@Starsam80
Copy link
Contributor

@Starsam80 Starsam80 commented Jan 1, 2017

I noticed this too, and when I looked more at it, it seems like cloudflare is at fault here. I suggest using cookies for now.

@kaithar
Copy link

@kaithar kaithar commented Jan 1, 2017

Glad it's not just me then.

@TheUnspeakableh
Copy link

@TheUnspeakableh TheUnspeakableh commented Jan 1, 2017

Same issue here.

@braydenm303
Copy link

@braydenm303 braydenm303 commented Jan 2, 2017

Including cookies isn't allowing me to pass the cloudflare check. (The cf_clearance cookie is in my cookie file.)

@Starsam80
Copy link
Contributor

@Starsam80 Starsam80 commented Jan 2, 2017

@braydenm303 Last time I checked, cloudflare uses more than just one cookie. These are the cookies that I have and it works for me: __cfduid, __qca, c_d, c_userid, c_userkey, c_visitor, cf_clearance, bm_last_load_status, ki_r, ki_t

@braydenm303
Copy link

@braydenm303 braydenm303 commented Jan 2, 2017

@Starsam80 Hmm. Right now, I have __cfduid, c_d, c_userid, c_userkey, c_visitor, cf_clearance, and it is not working.

@mia-0
Copy link

@mia-0 mia-0 commented Jan 2, 2017

Add all of them and maybe also make your user agent string match your browser’s.

@braydenm303
Copy link

@braydenm303 braydenm303 commented Jan 2, 2017

Setting the user agent string worked. (FYI, cookies and agent string were from a Chromium-based browser (Opera beta 43.0) in case anyone else has a similar problem.)

@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Jan 2, 2017

How are you all extracting cookies? Plugins?

@mia-0
Copy link

@mia-0 mia-0 commented Jan 2, 2017

Maybe it’s time to integrate cf-specific workarounds because cf decided to treat users like crap?
This is going to be nasty, but there’s https://github.com/Anorov/cloudflare-scrape for example.

@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Jan 2, 2017

cfscrape is indeed an awesome and handy tool. I've replaced requests in some of my projects with cfscrape. But, cfscrape only works for those simple browser checks. If there's a captcha, it'll fail.

@Crank666
Copy link

@Crank666 Crank666 commented Jan 4, 2017

Could someone pls show me a working command? Seems im too stupid to get the cookie file / user agent thing working :/ thanks in advance!!

@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Jan 4, 2017

Find your useragent : http://www.esolutions.se/whatsmyinfo
Grab/Export browser cookies via any plugin like 'export cookies' (mozilla)

Pass that file with the argument --cookies fileName.txt and then your user agent as --user-agent userAgent.

@Crank666
Copy link

@Crank666 Crank666 commented Jan 4, 2017

thx! sadly doesn´t work for me:
"youtube-dl http://www.crunchyroll.com/monster-strike -u *** -p *** -f best --write-sub --sub-lang deDE --sub-format ass --playlist-start 28 --cookies cookies.txt --user-agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
where do you put the cookies.txt in? (atm included in the main youtube-dl dir)

@jakepbrown27
Copy link

@jakepbrown27 jakepbrown27 commented Jan 4, 2017

@Crank666 Hi there, make sure you browse to crunchyroll after you install the plugin, after you do that the top entries should be crunchyroll cookies in either the .txt or the plugin.

@jakepbrown27
Copy link

@jakepbrown27 jakepbrown27 commented Jan 4, 2017

image
Should look something like this

@Crank666
Copy link

@Crank666 Crank666 commented Jan 4, 2017

'# HTTP Cookie File for domains related to crunchyroll.com.
'# To download this content click here, press return (Mac only) or copy and paste into a cookies.txt file.
'# It can be then used by wget and other standard complaint tools.
'# Example: wget -x --load-cookies cookies.txt http://www.crunchyroll.com/space-patrol-luluco
'#
.crunchyroll.com TRUE / FALSE __cfduid
.crunchyroll.com TRUE / FALSE c_visitor
.crunchyroll.com TRUE / FALSE __qca
www.crunchyroll.com FALSE / FALSE 0 PHPSESSID
.crunchyroll.com TRUE / FALSE c_market_ref
www.crunchyroll.com FALSE / FALSE ki_t
www.crunchyroll.com FALSE / FALSE ki_r
.www.crunchyroll.com TRUE / FALSE __ar_v4
.crunchyroll.com TRUE / FALSE cf_clearance
.crunchyroll.com TRUE / FALSE 0 sess_id
.crunchyroll.com TRUE / FALSE c_userid
.crunchyroll.com TRUE / FALSE c_userkey
.crunchyroll.com TRUE / FALSE c_d

.crunchyroll.com TRUE / FALSE _gat
.crunchyroll.com TRUE / FALSE _ga

only the above mentioned "bm_last_load_status" is missing...

-> after i run the youtube-dl command, the cookies.txt is replaced with this:
'# Netscape HTTP Cookie File
'# http://curl.haxx.se/rfc/cookie_spec.html
'# This is a generated file! Do not edit.

.crunchyroll.com TRUE / FALSE __cfduid
.crunchyroll.com TRUE / FALSE __qca
.crunchyroll.com TRUE / FALSE _ga
.crunchyroll.com TRUE / FALSE _gat
.crunchyroll.com TRUE / FALSE c_d
.crunchyroll.com TRUE / FALSE c_market_ref
.crunchyroll.com TRUE / FALSE c_userid
.crunchyroll.com TRUE / FALSE c_userkey
.crunchyroll.com TRUE / FALSE c_visitor
.crunchyroll.com TRUE / FALSE cf_clearance
.www.crunchyroll.com TRUE / FALSE __ar_v4
www.crunchyroll.com FALSE / FALSE ki_r
www.crunchyroll.com FALSE / FALSE ki_t

@Ugrend
Copy link

@Ugrend Ugrend commented Jan 5, 2017

@Crank666 If you still haven't got it working, remove the username/password parameters and it should work (it will auth with the cookie file)

@madrasile
Copy link

@madrasile madrasile commented Jan 5, 2017

Hi,

You have to check these things :

  • You're using the same user agent in yt-dl than the one you used to generate cookies
  • You're using the same IP address than the compute where you generated cookies

Also what's your error ?

@HigHendHd
Copy link

@HigHendHd HigHendHd commented Jan 5, 2017

This unfortunately does not work for me either. The command + output is the following:

youtube-dl http://www.crunchyroll.com/fullmetal-alchemist-brotherhood/episode-1-fullmetal-alchemist-718109 --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" --write-sub --sub-lang enUS -F --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://www.crunchyroll.com/fullmetal-alchemist-brotherhood/episode-1-fullmetal-alchemist-718109', u'--user-agent', u'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0', u'--write-sub', u'--sub-lang', u'enUS', u'-F', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.05
[debug] Python version 2.7.12 - Linux-4.4.0-57-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 3.2.2-static, ffprobe 3.2.2-static
[debug] Proxy map: {}
[Crunchyroll] 718109: Downloading webpage
[Crunchyroll] 718109: Downloading media info for 360p
[Crunchyroll] 718109: Downloading m3u8 information
[Crunchyroll] 718109: Downloading media info for 480p
[Crunchyroll] 718109: Downloading m3u8 information
[Crunchyroll] 718109: Downloading media info
[Crunchyroll] 718109: Downloading subtitles for English (US)
[info] Available formats for 718109:
format code extension resolution note
hls-meta-0 mp4 multiple Quality selection URL
hls-meta-1 mp4 multiple Quality selection URL
hls-474 mp4 640x360 474k , avc1.4d401e, 23.98fps, mp4a.40.2
hls-704-0 mp4 848x480 704k , avc1.4d401e, 23.98fps, mp4a.40.2
hls-704-1 mp4 848x480 704k , avc1.4d401e, 23.98fps, mp4a.40.2
hls-704-2 mp4 848x480 704k , avc1.4d401e, 23.98fps, mp4a.40.2 (best)

My Cookies File looks like this:

Netscape HTTP Cookie File

http://www.netscape.com/newsref/std/cookie_spec.html

This is a generated file! Do not edit.

.crunchyroll.com TRUE / FALSE 1515190731 __cfduid *****
.crunchyroll.com TRUE / FALSE 1499206731 c_visitor *****
www.crunchyroll.com FALSE / FALSE 1483657200 blockTrack *****
.crunchyroll.com TRUE / FALSE 1484263145 cf_clearance *****
.crunchyroll.com TRUE / FALSE 0 sess_id *****
www.crunchyroll.com FALSE / FALSE 0 PHPSESSID *****
.crunchyroll.com TRUE / FALSE 1484864355 c_userid *****
.crunchyroll.com TRUE / FALSE 1484864355 c_userkey *****
.crunchyroll.com TRUE / FALSE 1484864355 c_d *****

@Crank666
Copy link

@Crank666 Crank666 commented Jan 6, 2017

@Ugrend worked perfectly! THX!
@HigHendHd i only used the cookie file - just type "youtube-dl http://www.crunchyroll.com/fullmetal-alchemist-brotherhood/episode-1-fullmetal-alchemist-718109 --write-sub --sub-lang enUS -F --cookies FILE.txt"
@madrasile i used the same ip and browser (tryed with FF and Chrome) - without login info it works perfectly (ugrends post)

@HigHendHd
Copy link

@HigHendHd HigHendHd commented Jan 6, 2017

@Crank666 Without the useragent it is working, thanks!

@HigHendHd
Copy link

@HigHendHd HigHendHd commented Jan 8, 2017

It seems the whole crunchyroll page is now protected with cloudflare and access is not possible at all anymore.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Jan 8, 2017

It seems the whole crunchyroll page is now protected with cloudflare and access is not possible at all anymore.

Not that bad :)

$ youtube-dl -v "http://www.crunchyroll.com/yuri-on-ice/episode-12-gotta-super-super-supercharge-it-grand-prix-final-free-skate-721643"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.crunchyroll.com/yuri-on-ice/episode-12-gotta-super-super-supercharge-it-grand-prix-final-free-skate-721643']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.05
[debug] Git HEAD: 121920114
[debug] Python version 3.6.0 - Linux-4.9.0-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] 721643: Downloading webpage
[Crunchyroll] 721643: Downloading media info for 360p
[Crunchyroll] 721643: Downloading m3u8 information
[Crunchyroll] 721643: Downloading media info for 480p
[Crunchyroll] 721643: Downloading m3u8 information
[Crunchyroll] 721643: Downloading media info
[debug] Invoking downloader on 'https://v.vrv.co/evs/assets/bac79dfc373e60297f4ca9663c068b99_3047603.mp4/index-v1-a1.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly92LnZydi5jby9ldnMvYXNzZXRzL2JhYzc5ZGZjMzczZTYwMjk3ZjRjYTk2NjNjMDY4Yjk5XzMwNDc2MDMubXA0LyoiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE0ODM4OTc3NTV9fX1dfQ__&Signature=bxseOne2fg3aohncNiwcLPSEotYHKgAvStb9SmUT38iDv9I~2nHL2M3b36oftG5eJPqvmgkLU3-zN4DD3Car1nkbduY2abWA8R3pfB~CCDgwvOezAR~kMxeKWW83P3jEw80vucgRCPoy00rf9~shY6QYXaXjSpQWrzgLGJC7Si-72yr26HMVgCWBEGEBYpHJ9Mwvcs-l9YCs8mAE7a81Q64O2oXoXv~glVXBhPFxHUYUkBTqGXcojdGCpjhuWSfRrBE3s0MWrUEgaP164MzJHZlO2IZEfoQLapxF4Dj9CwCfVuOp7STeJCYt3uolUzST~wBx~mndjgfarDVPzihoPw__&Key-Pair-Id=APKAJKQQ2INNHTYFB44A'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 142
[download] Destination: Yuri!!! On ICE Episode 12 – Gotta Super-Super-Supercharge It!!! Grand Prix Final Free Skate-721643.mp4
[download]   4.2% of ~66.93MiB at 1000.45KiB/s ETA 04:36^CException ignored in: <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('140.112.28.103', 50288), raddr=('54.192.147.75', 443)>
ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('140.112.28.103', 50288), raddr=('54.192.147.75', 443)>

ERROR: Interrupted by user
Exception ignored in: <_io.FileIO name='Yuri!!! On ICE Episode 12 – Gotta Super-Super-Supercharge It!!! Grand Prix Final Free Skate-721643.mp4.part' mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.FileIO name='Yuri!!! On ICE Episode 12 – Gotta Super-Super-Supercharge It!!! Grand Prix Final Free Skate-721643.mp4.part' mode='wb' closefd=True>
@madrasile
Copy link

@madrasile madrasile commented Jan 8, 2017

It's still working for me to download .flv but now I got error 500 for downloading Subtitles, think it's a new issue and need to be fixed in a new version of youtube-dl

Nyanko Days Épisode 1 – Moi et mes chats adorés-728655.flv
[Crunchyroll] Downloading login page
[Crunchyroll] 728655: Downloading webpage
[Crunchyroll] 728655: Downloading media info for 360p
[Crunchyroll] 728655: Downloading media info for 480p
[Crunchyroll] 728655: Downloading media info for 720p
[Crunchyroll] 728655: Downloading media info for 1080p
ERROR: Unable to download XML: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[Crunchyroll] Downloading login page
[Crunchyroll] 728655: Downloading webpage
[Crunchyroll] 728655: Downloading media info for 360p
[Crunchyroll] 728655: Downloading media info for 480p
[Crunchyroll] 728655: Downloading media info for 720p
[Crunchyroll] 728655: Downloading media info for 1080p
[Crunchyroll] 728655: Downloading media info
[download] Destination: Nyanko Days Épisode 1 – Moi et mes chats adorés-728655.flv
[download]  55.1% of ~36.70MiB at 704.22KiB/s ETA 00:35

I'm gonna look for it tomorrow.

EDIT : Here is the error


ERROR: Unable to download XML: HTTP Error 500: Internal Server Error (caused by HTTPError()); please
report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 406, in _request_webpage
    return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2001, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
@mia-0
Copy link

@mia-0 mia-0 commented Jan 8, 2017

@madrasile
Copy link

@madrasile madrasile commented Jan 8, 2017

@lachs0r
It seems to be not a cookie problem for me for @HigHendHd problem, the user-agent should fix his problem.

Gonna try to refresh the cookie and also test it under windows for the subs.

@mia-0
Copy link

@mia-0 mia-0 commented Jan 8, 2017

@kaithar
Copy link

@kaithar kaithar commented Nov 10, 2017

Of course it makes sense... if you're using the site as originally intended, you HAVE to access a html page before the API is accessed.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Nov 11, 2017

APIs can be accessed from apps in other devices and environments(even from the CLI like in the case of Amazon AWS), and it doesn't make sense to include or redirect the user to a browser just to pass the cf anti-bot page.
anyway you're discussion is outside the subject of the issue, the issue was about finding a way to avoid the 503 error.

@ytdl-org ytdl-org locked and limited conversation to collaborators Nov 11, 2017
@ytdl-org ytdl-org unlocked this conversation Nov 12, 2017
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Nov 12, 2017

Still does not work:

> py -3.6 .\youtube_dl\__main__.py http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513 -n -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513', '-n', '-v']
[debug] Encodings: locale cp1251, fs utf-8, out utf-8, pref cp1251
[debug] youtube-dl version 2017.11.06
[debug] Git HEAD: 5fc12b9
[debug] Python version 3.6.2 - Windows-10-10.0.10240-SP0
[debug] exe versions: ffmpeg N-85653-gb4330a0, ffprobe N-85653-gb4330a0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] Logging in
[Crunchyroll] 645513: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by <HTTPError 503: 'Service Temporarily Unavailable'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Dev\youtube-dl\master\youtube_dl\extractor\common.py", line 506, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 2195, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python\Python36\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Python\Python36\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python\Python36\lib\urllib\request.py", line 564, in error
    result = self._call_chain(*args)
  File "C:\Python\Python36\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Python\Python36\lib\urllib\request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python\Python36\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Python\Python36\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python\Python36\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)
  File "C:\Python\Python36\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Python\Python36\lib\urllib\request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
@dstftw dstftw reopened this Nov 12, 2017
@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Nov 12, 2017

@dstftw I'd recommend checking out "cfscrape" for bypassing CloudFlare. I've been using it and it'll work with requests. You need NodeJS though.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Nov 12, 2017

See #11730 (comment) for my previous comment about cfscrape. Or we can add an option --use-insecure-features. Not sure if it's a good idea.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Nov 12, 2017

Oh, cfscrape now uses Node.js's vm feature. That's much better and we can revisit that :)

@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Nov 12, 2017

That's old... cfscrape doesn't use js2py anymore, as he ported it to "NodeJS". Check the Dependecies Section of Cfscrape.

@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Nov 12, 2017

Yep. Althought, people will need to have NodeJS installed. So, maybe you can pass an option like "--cf-bypass", which will enable cfscrape and hence, NodeJS? Or you can keep it on by default.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Nov 12, 2017

note that now there is only one request that has the problem of cf anti-bot page, i think that the extraction of the needed metadata can be done using other methods(use of Mobile API).

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Nov 12, 2017

Imposing restrictions on the mobile APIs is simpler. They can write complex HMAC codes in C, obfuscate it and strip all the useful debug info from shared libraries. That's a common practice in Chinese video sharing websites.

Is there a crunchyroll staff watching this ticket? :)

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Nov 12, 2017

Imposing restrictions on the mobile APIs is simpler. They can write complex HMAC codes in C, obfuscate it and strip all the useful debug info from shared libraries. That's a common practice in Chinese video sharing websites.

there are similar ways to achieve what Android NDK can do for apps on the web:

  • flash player(cross bridge) like the case of Iqiyi(flash is going to be absolute anyway in 2020).
  • JavaScript(emscripten) like in the case of bilibili.

but it's not unbreakable, it just needs more work, even the some of the common DRM schemes are already broken.

@Xonshiz
Copy link
Author

@Xonshiz Xonshiz commented Nov 13, 2017

I still think utilizing cfscrape would be easiest and fastest way to solve this xD!

@kaithar
Copy link

@kaithar kaithar commented Nov 13, 2017

@remitamine ffs, no

APIs can be accessed from apps in other devices and environments(even from the CLI like in the case of Amazon AWS), and it doesn't make sense to include or redirect the user to a browser just to pass the cf anti-bot page.
anyway you're discussion is outside the subject of the issue, the issue was about finding a way to avoid the 503 error.

This is not a public API. It's private and undocumented, intended for the purpose of funimation's site only. Thus your argument doesn't apply.
We're not talking about general purpose APIs, which are intended for non-browser use. Honestly this makes me even more suspicious that you haven't bypassed the 503 properly all.

And in general, see #11572 (comment) for the method I've been using to keep CR extractor working for some time now.

@BooDoo
Copy link

@BooDoo BooDoo commented Dec 16, 2017

Even with --cookies and --user-agent flags I was having no luck with a cookies.txt exported using the Firefox "cookies.txt" add-on.

I did a Find+Replace for #HttpOnly_ in the cookies file and stripped those out, now it's working perfectly. Hope this helps someone who got here looking for a solution.

@kueller
Copy link

@kueller kueller commented Dec 23, 2017

I'm having trouble with this too as of late. It had been working fine until recently just passing the cookies file in. I've tried with and without user agent, with and without usename/password. I've used the new cookies exporter with Firefox's update and removed #HttpOnly_ sections. Also tried the cfscrape method. Nothing is logging me in properly. I do have a premium account.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Dec 24, 2017

@kueller you mean #11730 doesn't work either!?

@kueller
Copy link

@kueller kueller commented Dec 24, 2017

It seems to get me past Cloudflare, but authenticating beyond that fails.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Dec 24, 2017

Could you paste verbose logs?

@kueller
Copy link

@kueller kueller commented Dec 27, 2017

Quick example of a premium-only video. Fresh cookies.txt with me logged in. Lack of a 503 leads me to believe it's getting past the Cloudflare layer.

$ youtube-dl --verbose --cookies cookies.txt http://www.crunchyroll.com/march-comes-in-like-a-lion/episode-33-chapter-66-where-the-sun-shines-chapter-67-small-world-754071
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'--cookies', u'cookies.txt', u'http://www.crunchyroll.com/march-comes-in-like-a-lion/episode-33-chapter-66-where-the-sun-shines-chapter-67-small-world-754071']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.12.23
[debug] Python version 2.7.14 - Linux-4.14.3-300.fc27.x86_64-x86_64-with-fedora-27-Twenty_Seven
[debug] exe versions: ffmpeg 3.3.5, ffprobe 3.3.5, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] 754071: Downloading webpage
ERROR: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 784, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 438, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python2.7/site-packages/youtube_dl/extractor/crunchyroll.py", line 426, in _real_extract
    raise ExtractorError(note_m)
ExtractorError: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Dec 27, 2017

@kueller: How about using --username and --password?

@Crank666
Copy link

@Crank666 Crank666 commented Dec 28, 2017

"youtube-dl http://www.crunchyroll.com/monster-strike/episode-2-the-karmic-fist-of-wrath-734365 -f best --write-sub --sub-lang deDE --sub-format ass --cookies cookies.txt --download-archive archive.txt"
works fine for me -> i use "cookies.txt" addon for chrome (Link)...

@kueller
Copy link

@kueller kueller commented Dec 28, 2017

Huh, ok. So I tried that Chrome extension and it works. I've been using a new cookie exporter from Firefox (after the recent update) and despite the cookies.txt looking valid it doesn't work. Whatever the reason at least I know it's not a problem here. Thank you for the help.

@Crank666
Copy link

@Crank666 Crank666 commented Dec 28, 2017

you´re welcome =) (had the same problems with FF - now its only second choice browser :( )

@remitamine remitamine closed this in dca0e00 Mar 2, 2019
@remitamine remitamine reopened this Mar 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.