Download from funk.net fails with HTTP Error 403: Forbidden / HTTP Error 400: Bad Request

[yeeeha]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.03.08
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

$ youtube-dl --verbose  https://www.funk.net/channel/doctor-who-1164/journeys-end-ov-515545/doctor-who-staffel-4-ov-2473

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.funk.net/channel/doctor-who-1164/journeys-end-ov-515545/doctor-who-staffel-4-ov-2473']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.08
[debug] Python version 2.7.12 (CPython) - Linux-4.15.0-88-generic-x86_64-with-LinuxMint-18.1-serena
[debug] exe versions: ffmpeg 2.8.15, ffprobe 2.8.15, rtmpdump 2.4
[debug] Proxy map: {}
[Funk] 515545: Downloading JSON metadata
[Nexx] 515545: Downloading JSON metadata
[Nexx] 515545: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[Nexx] 515545: Downloading MPD manifest
WARNING: Failed to download MPD manifest: HTTP Error 403: Forbidden
[Nexx] 515545: Downloading ISM manifest
WARNING: Failed to download ISM manifest: HTTP Error 403: Forbidden
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'http://nx-d09.akamaized.net/06be6b49-c771-4f80-9ceb-a658ebc5e751/515545_src_1920x1080_6000.mp4'
ERROR: unable to download video data: HTTP Error 400: Bad Request
Traceback (most recent call last):
  File "/home/yee/Develop/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 1926, in process_info
    success = dl(filename, info_dict)
  File "/home/yee/Develop/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 1865, in dl
    return fd.download(name, info)
  File "/home/yee/Develop/youtube-dl/youtube-dl/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/home/yee/Develop/youtube-dl/youtube-dl/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/home/yee/Develop/youtube-dl/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/yee/Develop/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: Bad Request

Description

The video https://www.funk.net/channel/doctor-who-1164/journeys-end-ov-515545/doctor-who-staffel-4-ov-2473 plays in a browser (Firefox) without problems, but youtube-dl fails to download m3u8 information, MPD & ISM manifests with HTTP Error 403: Forbidden.

About 3 weeks ago I didn't experience any problems downloading videos from this site. Additionally, the problem only seems to occur when the location is nx-d09.akamaized.net like

[debug] Invoking downloader on u'http://nx-d09.akamaized.net/06be6b49-c771-4f80-9ceb-a658ebc5e751/515545_src_1920x1080_6000.mp4'
ERROR: unable to download video data: HTTP Error 400: Bad Request

Other videos, e.g. https://www.funk.net/channel/true-demon-12035/camping-im-wald-nimmt-fuerchterliches-ende-01-1616275/true-demon-alle-folgen-4971 from a different location funk-01.akamaized.net can still be downloaded:

$ youtube-dl --verbose https://www.funk.net/channel/true-demon-12035/camping-im-wald-nimmt-fuerchterliches-ende-01-1616275/true-demon-alle-folgen-4971

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.funk.net/channel/true-demon-12035/camping-im-wald-nimmt-fuerchterliches-ende-01-1616275/true-demon-alle-folgen-4971']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.08
[debug] Python version 2.7.12 (CPython) - Linux-4.15.0-88-generic-x86_64-with-LinuxMint-18.1-serena
[debug] exe versions: ffmpeg 2.8.15, ffprobe 2.8.15, rtmpdump 2.4
[debug] Proxy map: {}
[Funk] 1616275: Downloading JSON metadata
[Nexx] 1616275: Downloading JSON metadata
[Nexx] 1616275: Downloading m3u8 information
[Nexx] 1616275: Downloading MPD manifest
[Nexx] 1616275: Downloading ISM manifest
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'http://funk-01.akamaized.net/098fec4a-c9a8-446e-bcb0-d4ae19f2612c/1616275_src.ism/QualityLevels(5978504)/Manifest(video,format=m3u8-aapl)'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 120
[download] Destination: Camping im Wald nimmt fürchterliches Ende #01-1616275.fazure-hls-6257.mp4
[download]   2.9% of ~567.00MiB at  5.55MiB/s ETA 01:46^C
ERROR: Interrupted by user

Thank you for your work.

[yeeeha]

Meanwhile I had time to do some debugging. The cause of the above problem is that funk.net has started to use Token Authentication (https://learn.akamai.com/en-us/webhelp/adaptive-media-delivery/adaptive-media-delivery-implementation-guide/GUID-740A5744-09DA-4763-A652-CBDDE5CC0133.html).
While the Nexx Extractor is perfectly able to deal with Token Authentication it just fails to apply it, partly because the protection token isn't stored in the expected token field within the JSON metadata, but in the fields tokenHLS and tokenDASH respectively.
I've prepared a fix and I'm going to create a pull request.

However, now I've encountered another issue: ffmpeg fails to merge video and audio files:

ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
...
[mp4 @ 0x1a27b80] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
...

I think fixing this should be straightforward, but I don't know if the fix should go to the same merge request?

[yeeeha]

I've got a fix for the ffmpeg issue (Malformed AAC bitstream) as well. Because it's only loosely related to the download problem, I've committed it to a different branch and I'll create a second pull request.
However, I can combine the two fixes if you prefer.

[remitamine]

@yeeeha update FFmpeg.

[yeeeha]

@remitamine: You're right. After updating ffmpeg to version 4.2.2 the Malformed AAC bitstream error is gone, so my workaround is unnecessary. Thank you.

Please still consider my pull request #24488 fixing the original download issue.

[yeeeha]

My pull request #24488 won't be accepted. Sorry to say it, but trying to collaborate on this project really gives you a PITA...
If someone's interested in the fix: You may have a look at https://github.com/yeeeha/youtube-dl/tree/fix-24309
I'm out. Thank you and goodbye.

[lukasfink1]

@yeeeha Thank you for your work. It seems astonishing hard to at least just point out the line numbers where the presumed code duplication might be.