[nrk:series] No video formats found / 404 not found

[YumekuiNeru]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.03.01
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

$ youtube-dl 'https://tv.nrk.no/serie/harald-eia-presenterer-saann-er-norge/sesong/1/episode/1/avspiller' --verbose
[debug] System config: []
[debug] User config: ['--buffer-size', '16K']
[debug] Custom config: []
[debug] Command-line args: ['https://tv.nrk.no/serie/harald-eia-presenterer-saann-er-norge/sesong/1/episode/1/avspiller', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.01
[debug] Python version 3.6.10 (CPython) - Linux-5.5.5-gentoo-pix-x86_64-Intel-R-_Core-TM-_i7-6700K_CPU_@_4.00GHz-with-gentoo-2.7
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[NRKTVEpisode] harald-eia-presenterer-saann-er-norge/sesong/1/episode/1: Downloading webpage
[debug] Using fake IP 84.212.51.255 (NO) as X-Forwarded-For.
[NRK] KMTE31000118: Downloading mediaelement JSON
[NRK] KMTE31000118: Downloading f4m manifest
WARNING: Unable to download f4m manifest: HTTP Error 404: Not Found
[NRK] KMTE31000118: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 404: Not Found
[NRK] KMTE31000118: Downloading f4m manifest
WARNING: Unable to download f4m manifest: HTTP Error 404: Not Found
[NRK] KMTE31000118: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 404: Not Found
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib64/python3.6/site-packages/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib64/python3.6/site-packages/youtube_dl/extractor/nrk.py", line 96, in _real_extract
    self._sort_formats(formats)
  File "/usr/lib64/python3.6/site-packages/youtube_dl/extractor/common.py", line 1327, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I see mentions of a fargerik-psapi.nrk.no in some dumped files.

[Roxedus]

Spent a few days looking into it, for me it looks like Akamai now offers a new way to serve manifests, which common._extract_akamai_formats cannot find (I had no luck either with Developer console in Chrome).

youtube-dl/youtube_dl/extractor/common.py

Lines 2562 to 2583 in 7947a1f

	def _extract_akamai_formats(self, manifest_url, video_id, hosts={}):
	formats = []
	hdcore_sign = 'hdcore=3.7.0'
	f4m_url = re.sub(r'(https?://[^/]+)/i/', r'\1/z/', manifest_url).replace('/master.m3u8', '/manifest.f4m')
	hds_host = hosts.get('hds')
	if hds_host:
	f4m_url = re.sub(r'(https?://)[^/]+', r'\1' + hds_host, f4m_url)
	if 'hdcore=' not in f4m_url:
	f4m_url += ('&' if '?' in f4m_url else '?') + hdcore_sign
	f4m_formats = self._extract_f4m_formats(
	f4m_url, video_id, f4m_id='hds', fatal=False)
	for entry in f4m_formats:
	entry.update({'extra_param_to_segment_url': hdcore_sign})
	formats.extend(f4m_formats)
	m3u8_url = re.sub(r'(https?://[^/]+)/z/', r'\1/i/', manifest_url).replace('/manifest.f4m', '/master.m3u8')
	hls_host = hosts.get('hls')
	if hls_host:
	m3u8_url = re.sub(r'(https?://)[^/]+', r'\1' + hls_host, m3u8_url)
	formats.extend(self._extract_m3u8_formats(
	m3u8_url, video_id, 'mp4', 'm3u8_native',
	m3u8_id='hls', fatal=False))
	return formats

This show is the only one I found on NRK that behaves like this, I have checked other shows with similar traits, like geo-block (this one shouldn't be), and same production year (2020).

fargerik-psapi.nrk.no is only used to personalize the site, looking at the indexed docs for the api.

[YumekuiNeru]

If it is useful, including the part that specifies the episode in the URL makes the download work for episodes 3-8, but fails for 1+2

[Roxedus]

It didn't exactly help me, but I did manage to get it to download those two episodes. However I cant find any value in the api to differentiate the working ones, vs the "broken" ones.

It looks like NRK did something wrong on the two first episodes, which also can be confirmed when looking at the data object created in nrk.py#34, and the data.mediaAssetsOnDemand.0.hlsUrl value which returns the same url as the one common._extract_akamai_formats function sends to _extract_m3u8_formats, which does not work for the two episodes, but the others.

The URLs from the CDN looks to be split on availability, worldwide and Norway, looking at the start of the url.

http://*.akamaihd.net/i/no/<...>/master.m3u8 - Norway

http://*.akamaihd.net/i/wo/<...>/master.m3u8 - Worldwide

Adding a regular expression to replace wo to no, fixes it for these two episodes, but breaks anything that's available worldwide, in other words, the rest of the season. I have not found any indicator to determine if this replacement is necessary before attempting to download. Any "fix" by me will be crude, and depend on a try/expect kinda deal.

To get the two episodes I added a new line after line 62 in nrk.py.
asset_url = re.sub(r'(https?://[^/]+)/z/wo/', r'\1/z/no/', asset_url)

Either this is a external-bugs or a wont-fix, unless someone well more versed in the project can figure out a more elegant way.