Crunchyroll: Unable To Download Webpage: HTTP Error 403

[AldoDominguez]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.06.16.1
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'--write-sub', u'--sub-lang', u'enUS', u'--embed-subs', u'https://www.crunchyroll.com/the-god-of-high-school']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.06.16.1
[debug] Python version 2.7.16 (CPython) - Darwin-19.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[crunchyroll:playlist] Downloading login page
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Description

Running into error message ( HTTP Error 403 ) on Crunchyroll site only, I tried to clear the youtube-dl cache with the following command: --rm-cache-dir but it still comes up with the same message afterwards, I made sure everything was up to date and the site is working. I don't see an open case for this issue so I apologize in advance if I missed something.

[TommoCrabb]

I'm having exactly the same problem.

[muneebmahmed]

Since #!/usr/bin/env python will search for python, this means the executable found will be Python 2.7 on a Mac instead of Python 3. Using Python3 should probably solve your issue. Some options:

Make python point to Python 3

Just make the first python executable in your PATH point to Python3 instead of the default Python2 in /usr/bin. Since you're on macOS Catalina, you should already have it installed, but in case anyone else is reading, it can be installed with brew or some other option. Then create a symlink in /usr/local/bin or somewhere else in your PATH that's before /usr/bin/python:

ln -s `which python3` /usr/local/bin/python

Note this might break other applications that depend on python being 2.7.

Use Python 3 as the interpreter by default

Instead of searching for python, search for `python3. Clone the repo, and change the following line in the Makefile:

youtube-dl/Makefile

Line 12 in a115e07

PYTHON ?= /usr/bin/env python

to

PYTHON ?= /usr/bin/env python3

(or the path to your python3 executable if you have a nonstandard install) then make should create the youtube-dl executable that you can place somewhere in your PATH.
If you don't want to clone the repo, I think you could probably also just edit the first line of the youtube-dl archive that you already have, using an editor that has a binary mode, like vim -b /usr/local/bin/youtube-dl -c '1s/python/python3/' -c wq.

Install with pip3

Since pip3 is a Python3 tool, it should probably work if you just install it with that:

sudo -H pip3 install --upgrade youtube-dl

I haven't tried installing with pip3 though.

[AldoDominguez]

@muneebmahmed thank you so much for all the info, I tried the steps you listed and I believe it was all done correctly but Im still getting the 403 error withe following log. Any ideas of what else it could be or maybe if I missed something else?

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', '--write-sub', '--sub-lang', 'enUS', '--embed-subs', '--verbose', 'https://www.crunchyroll.com/the-god-of-high-school\u2028']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.06.16.1
[debug] Python version 3.8.4 (CPython) - macOS-10.15.5-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[crunchyroll:playlist] Downloading login page
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.8/site-packages/youtube_dl/YoutubeDL.py", line 2238, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/local/Cellar/python@3.8/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/local/Cellar/python@3.8/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/local/Cellar/python@3.8/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/local/Cellar/python@3.8/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python@3.8/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

[muneebmahmed]

@AldoDominguez Oh, I didn't see that you were passing in username and password on the command line. #25481 appears to be a related issue... Instead of supplying username and password from the command line, have you tried logging in with a web browser, exporting your cookies to a cookies.txt file and then passing it in with the --cookies option?

From the discussion on the other issue it looks like the problem is that TLS 1.3 is needed, but I'm not sure why it works without credentials but not for the login page.

[AldoDominguez]

@muneebmahmed I've actually tried it without the credentials and I still run into the same issue, this page does have free videos to so even those first free episodes should download. 😢

[TommoCrabb]

@AldoDominguez I managed to solve my 403 error by upgrading from python 3.6 to 3.8; however, I see that you're already using 3.8, so obviously your problem is different to mine.

Now, this is just be a stab in the dark, but I noticed in your error message, that the URL ('https://www.crunchyroll.com/the-god-of-high-school\u2028') ends in "\u2028", which I believe is a new line character. Maybe this is the problem?