HTTPError 426: 'Upgrade Required' when downloading from CBC

[dazey3]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

• I'm reporting a bug unrelated to a specific site
• I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Hello! I was downloading from CBC just fine for a few days and everything was fine. Suddenly all my CBC downloads started failing with the following error:

yt-dlp -o "C:\Users\...\yt-dlp\%%(channel)s - %%(title)s.%%(ext)s" https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] the-fifth-estate/s41e09: Downloading JSON metadata
ERROR: [gem.cbc.ca] the-fifth-estate/s41e09: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

yt-dlp -U

Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• If using API, add 'verbose': True to YoutubeDL params instead
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt_dlp -vU -o "C:\<redacted>\yt-dlp\%%(channel)s - %%(title)s.%%(ext)s" https://gem.cbc.ca/media/the-fifth-estate/s41e09
[debug] Command-line config: ['-vU', '-o', 'C:\\<redacted>\\yt-dlp\\%%(channel)s - %%(title)s.%%(ext)s', 'https://gem.cbc.ca/media/the-fifth-estate/s41e09']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 68be95bd0
[debug] Python 3.10.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1s  1 Nov 2022)
[debug] exe versions: none
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1791 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[debug] Using fake IP 99.230.144.152 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] the-fifth-estate/s41e09: Downloading JSON metadata
ERROR: [gem.cbc.ca] the-fifth-estate/s41e09: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "C:\<redacted>\yt_dlp\extractor\cbc.py", line 352, in _real_extract
    video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "C:\<redacted>\yt_dlp\extractor\common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "C:\<redacted>\YoutubeDL.py", line 3743, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 563, in error
    return self._call_chain(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 426: Upgrade Required

[cagenuts]

Is this a bug with the yt-dlp extractor? Downloading 720p works (directly) but -F doesn't show the 1080p stream.

[makew0rld]

I can confirm this issue, I will look into it. Worst case I will be free to explore this in a couple weeks.

[cagenuts]

@makew0rld thank you, I for one will appreciate it.

[fbjon]

I also have this issue, the http traffic shows an interesting Upgrade header:

[gem.cbc.ca] murdoch-mysteries/s11e08: Downloading JSON metadata
send: b'GET /ott/cbc-api/v2/assets/murdoch-mysteries/s11e08 HTTP/1.1\r\nHost: services.radio-canada.ca\r\nX-Forwarded-For: 99.247.151.173\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.74 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nSec-Fetch-Mode: navigate\r\nAccept-Encoding: gzip, deflate, br\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 426 Upgrade Required\r\n'
header: Server: nginx/1.23.2
header: Content-Type: application/json
header: Content-Length: 2398
header: Expires: Fri, 07 Apr 2023 19:05:47 GMT
header: Cache-Control: max-age=0, no-cache, no-store
header: Pragma: no-cache
header: Date: Fri, 07 Apr 2023 19:05:47 GMT
header: Connection: close
header: Upgrade: 11.0.0
header: gem-api-version: 1.20.0
ERROR: [gem.cbc.ca] murdoch-mysteries/s11e08: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

[cagenuts]

So we need to upgrade the protocol sent from yt-dlp.

[makew0rld]

I will investigate what the browser client does, as that is what is best to copy. But when I actually visit the URL, while it does return HTTP code 426 and have that non-standard Upgrade header, all the expected JSON metadata is there. I suspect if youtube-dl just ignored the HTTP code things would continue working fine.

Will look more into this later as mentioned above.

[jo-nike]

@makew0rld I wanted to test your theory and was able to confirm that you are correct.

As a test I added expected_status = 426 here: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/common.py#L881, recompiled my executable and confirmed it worked.

Before:

redacted@redacted:/mnt/c/Users/redacted/Downloads$ ./yt-dlp https://gem.cbc.ca/media/plan-b/s01e03 -v
[debug] Command-line config: ['https://gem.cbc.ca/media/plan-b/s01e03', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.1
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Using fake IP 99.229.117.88 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/plan-b/s01e03
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
ERROR: [gem.cbc.ca] plan-b/s01e03: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "./yt-dlp/yt_dlp/extractor/common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "./yt-dlp/yt_dlp/extractor/cbc.py", line 349, in _real_extract
    video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
  File "./yt-dlp/yt_dlp/extractor/common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "./yt-dlp/yt_dlp/extractor/common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "./yt-dlp/yt_dlp/YoutubeDL.py", line 3742, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 426: Upgrade Required

After:

redacted@redacted:/mnt/d/Code/yt-dlp$ ./yt-dlp https://gem.cbc.ca/media/plan-b/s01e03 -v
[debug] Command-line config: ['https://gem.cbc.ca/media/plan-b/s01e03', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.1
[debug] Proxy map: {}
[debug] Loaded 1791 extractors
[debug] Using fake IP 99.235.152.163 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/plan-b/s01e03
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
[gem.cbc.ca] plan-b/s01e03: Downloading m3u8 information
[gem.cbc.ca] plan-b/s01e03: Downloading secret XML
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] plan-b/s01e03: Downloading 1 format(s): sec-1080+hls-audio-English
[debug] Invoking hlsnative downloader on "https://cbcrcott-gem.akamaized.net/9a7e7072-ad0a-4e25-b860-2a9f40c8761a/CBC_PLANB_CBC_S01E03.ism/QualityLevels(5999851)/Manifest(video,format=m3u8-aapl)"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 438
[download] Destination: Episode 3 [plan-b⧸s01e03].fsec-1080.mp4
[download] 100% of    1.88GiB in 00:01:58 at 16.24MiB/s
[debug] Invoking hlsnative downloader on "https://cbcrcott-gem.akamaized.net/9a7e7072-ad0a-4e25-b860-2a9f40c8761a/CBC_PLANB_CBC_S01E03.ism/QualityLevels(192000)/Manifest(audio_eng_aacl,format=m3u8-aapl,filter=desktop)"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 435
[download] Destination: Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a
[download] 100% of   62.86MiB in 00:00:39 at 1.57MiB/s
[debug] ffmpeg command line: ffprobe -show_streams 'file:Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a'
[Merger] Merging formats into "Episode 3 [plan-b⧸s01e03].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Episode 3 [plan-b⧸s01e03].fsec-1080.mp4' -i 'file:Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a' -c copy -map 0:v:0 -map 1:a:0 -bsf:a:0 aac_adtstoasc -movflags +faststart 'file:Episode 3 [plan-b⧸s01e03].temp.mp4'
Deleting original file Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a (pass -k to keep)
Deleting original file Episode 3 [plan-b⧸s01e03].fsec-1080.mp4 (pass -k to keep)

[jo-nike]

My above message wasn't necessarily a suggestion for a patch, was just a dirty confirmation of the theory.

This would be more gracious:

This line can be modified: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/cbc.py#L352

from:
video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)

to
video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id, expected_status=426)

[cagenuts]

@jo-nike Brilliant! Thanks man.

[delphs]

I update both my files and we're back in business! thanks a lot!!!

[jo-nike]

I update both my files and we're back in business! thanks a lot!!!

Be careful tho, updating common.py like I suggested in the first message could compromise other extractors. It wasn't meant to be a fix at all. Great that it work but look out for a fix in the future and update your code accordingly to avoid undesired results with other extractors.

[delphs]

I update both my files and we're back in business! thanks a lot!!!

Be careful tho, updating common.py like I suggested in the first message could compromise other extractors. It wasn't meant to be a fix at all. Great that it work but look out for a fix in the future and update your code accordingly to avoid undesired results with other extractors.

Completely understand! In my case I only use it for CBC GEM and nothing else.

[makew0rld]

Ok I can confirm the browser client is doing something different now, it no longer accesses the URL currently used by yt-dlp. It's good to see #6781, thanks @jo-nike. But I expect this method will break in the future when CBC removes that API. I would agree that this issue can be closed once #6781 is merged though.

[WolfganP]

Just confirming that I tried today with https://gem.cbc.ca/media/best-in-miniature/s02e01 and patch #6781 works ok.

[WolfganP]

Side question: I installed the patched extractor at ~/.config/yt-dlp/plugins/extractor/yt_dlp_plugins/cbc.py in another computer but yt-dlp still picks up the core cbc extractor and fails to parse the URL.

Did I missunderstand the readme plugins sideload section? TIA.

[bashonly]

~/.config/yt-dlp/plugins/extractor/yt_dlp_plugins/cbc.py
should be
~/.config/yt-dlp/plugins/cbc_plugin/yt_dlp_plugins/extractor/cbc.py

[WolfganP]

~/.config/yt-dlp/plugins/extractor/yt_dlp_plugins/cbc.py should be ~/.config/yt-dlp/plugins/cbc_plugin/yt_dlp_plugins/extractor/cbc.py

Perfect, thx a lot.