HTTP Error 403: Forbidden when trying to download playlists from https://learning.oreilly.com/

[lukahn]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2019.11.22
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

youtube-dl --verbose --cookies cookies "https://learning.oreilly.com/videos/hadoop-fundamentals-livelessons/9780133392838"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'--cookies', u'cookies', u'https://learning.oreilly.com/videos/hadoop-fundamentals-livelessons/9780133392838']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.11.22
[debug] Python version 2.7.16 (CPython) - Linux-5.0.0-27-generic-x86_64-with-Ubuntu-19.04-disco
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3
[debug] Proxy map: {}
[safari:course] 9780133392838: Downloading course JSON
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2237, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Description

When trying to download a playlist (such as the example above found in https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/safari.py), I get the message "HTTP Error 403: Forbidden". This happens in both Linux and Windows. The video requires an account (free is fine), so maybe that's part of the issue. There was a previous ticket to fix authentication here: #22184, but downloading that older build (https://github.com/ytdl-org/youtube-dl/releases/tag/2019.09.01) gives the same behaviour.

Downloading an individual video is fine, but just not a playlist.

Also of note is when downloading an individual video, if I use basic authentication (username/password), then I get the following error:

WARNING: Unable to download kaltura session JSON: HTTP Error 403: Forbidden

This doesn't happen when using cookies.

[lukahn]

I got it working now, thanks taking the time to confirm it.
For anyone else that finds this in the future (probably me), the problem was that my trial period had expired (even though I could still watch the videos on the website, the API functionality was restricted). I found the expiry here:
https://learning.oreilly.com/api/v1/

...
"user_type": "Expired",
...
"trial": {
"trial_expiration_date": "2019-05-14T01:22:59.14522Z",
"trial": true
},
...

The endpoint that youtube-dl hits was:
https://learning.oreilly.com/api/v1/book/9780133392838/?override_format=json
Whereby I saw:

{
"detail": "You do not have permission to perform this action."
}

I fixed it by creating a new trial account, and using that. I also used Firefox to get the cookies.txt file this time, which may also have helped.