Unable to download webpage: HTTP Error 403: Forbidden

[kuralex1]

Got the following error:

youtube-dl --username "user" --password "pass" --verbose --sleep-interval 120 "https://app.pluralsight.com/library/courses/microsoft-azure-microsoft-graph-app-service-integrating/"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', '--verbose', '--sleep-interval', '120', 'https://app.pluralsight.com/library/courses/microsoft-azure-microsoft-graph-app-service-integrating/']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: none
[debug] Proxy map: {}
[pluralsight:course] microsoft-azure-microsoft-graph-app-service-integrating: Downloading JSON metadata
[download] Downloading playlist: Microsoft Azure Developer: Integrating an App or Service with Microsoft Graph
[pluralsight:course] playlist Microsoft Azure Developer: Integrating an App or Service with Microsoft Graph: Collected 13 video ids (downloading 13 of them)
[download] Downloading video 1 of 13
[pluralsight] Downloading login page
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 627, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 2238, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

[robintown]

Have you read this?

[kuralex1]

thx, just tried with --rm-cache-dir the the same error:

d:\Alex\DownloadVideo>youtube-dl --username "user" --password "pass" --verbose --sleep-interval 120 --rm-cache-dir "https://app.pluralsight.com/library/courses/microsoft-azure-microsoft-graph-app-service-integrating/"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', '--verbose', '--sleep-interval', '120', '--rm-cache-dir', 'https://app.pluralsight.com/library/courses/microsoft-azure-microsoft-graph-app-service-integrating/']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: none
[debug] Proxy map: {}
Removing cache dir C:\Users\Alex/.cache\youtube-dl ..
[pluralsight:course] microsoft-azure-microsoft-graph-app-service-integrating: Downloading JSON metadata
[download] Downloading playlist: Microsoft Azure Developer: Integrating an App or Service with Microsoft Graph
[pluralsight:course] playlist Microsoft Azure Developer: Integrating an App or Service with Microsoft Graph: Collected 13 video ids (downloading 13 of them)
[download] Downloading video 1 of 13
[pluralsight] Downloading login page
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 627, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 2238, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

[munim]

YTDL version: 2020.03.24

Tried with referer header, min/max sleep interval and that doesn't work either.

youtube-dl --newline --username username --password passwork --ignore-config --add-header Referer:'https://app.pluralsight.com/library/courses/' --sleep-interval 30 --max-sleep-interval 120 --verbose -o "%(playlist_title)s/%(chapter_number)s - %(chapter)s/%(playlist_index)s-%(title)s.%(ext)s" "https://app.pluralsight.com/library/courses/angularjs-patterns-clean-code"

Error as shown below:

ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

[ddbb2017]

@kuralex1 change user agent manually. They probably started blocking by whatever user agent that is used by default. So add --user-agent "your-user-agent" command. You can pick some user agent from here.

[munim]

@kuralex1 change user agent manually. They probably started blocking by whatever user agent that is used by default. So add --user-agent "your-user-agent" command. You can pick some user agent from here.

@ddbb2017 Passing Chrome user agent doesn't work either.

[ddbb2017]

@munim it started working absolutely fine on my end after the user agent change. So if Chrome doesn't work, try some other ua.

overall this software is very quirky, so I'm sure it leaves a lot of traces for the video service to track. There may be other inconsistencies in it that I'm not aware of. I've been trying to coax it to work for just a few days so far.

[kuralex1]

Chrome is not working, with IE11 ua is fine.
--user-agent "Mozilla/5.0 CK={} (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"

Thanks a lot.

[ddbb2017]

@kuralex1 I tested it some more yesterday and there's a combination of the user agent, IP address and your downloading pattern that can cause this 403 error. For instance I was struggling to set a proper interval between videos.

The software is configured in a very weird way. It downloads the meta data for a video and then the waiting period, that is defined by --min-sleep-interval and --max-sleep-interval kicks in. So if you set it to a few minutes what happens is that the meta data is downloaded first, then this weird pause starts and then it downloads the video itself. As you can imagine the real player doesn't do that :) So I bet this pattern alone is enough to throw a red a flag and blacklist the IP or the account itself.

Plus there's no way to specify the pause between courses. Say if the pause between videos is 2-4 min, it'd be wise to have an option to specify a pause between courses, say 10-15 min or so. Or, ideally to base the pause between videos based on its play duration of the previous video.

In other words it's all doable. I think I will write my own script for it though. I tried posting all this here and there's some hotshot that closes issues without even considering them. Pretty fucked up in my view.

[Sudheerkunda]

--user-agent "Mozilla/5.0 CK={} (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"

can you show me the commands how can u entered