New ZDF-Mediathek broken

[retsifp]

• I've verified and I assure that I'm running youtube-dl 2016.10.26

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

$ youtube-dl -v "https://www.zdf.de/comedy/neo-magazin-mit-jan-boehmermann/neo-magazin-royale-vom-27-10-2016-102.html"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://www.zdf.de/comedy/neo-magazin-mit-jan-boehmermann/neo-magazin-royale-vom-27-10-2016-102.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.10.26
[debug] Python version 2.7.12+ - Linux-4.7.0-1-amd64-x86_64-with-debian-stretch-sid
[debug] exe versions: ffmpeg 3.1.4-1, ffprobe 3.1.4-1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] neo-magazin-royale-vom-27-10-2016-102: Requesting header
WARNING: Falling back on generic information extractor.
[generic] neo-magazin-royale-vom-27-10-2016-102: Downloading webpage
[generic] neo-magazin-royale-vom-27-10-2016-102: Extracting information
ERROR: Unsupported URL: https://www.zdf.de/comedy/neo-magazin-mit-jan-boehmermann/neo-magazin-royale-vom-27-10-2016-102.html
Traceback (most recent call last):
  File "/home/thomas/youtube-dl/youtube-dl/youtube_dl/extractor/generic.py", line 1634, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/home/thomas/youtube-dl/youtube-dl/youtube_dl/compat.py", line 2525, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/home/thomas/youtube-dl/youtube-dl/youtube_dl/compat.py", line 2514, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 141, column 162
Traceback (most recent call last):
  File "/home/thomas/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "/home/thomas/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 356, in extract
    return self._real_extract(url)
  File "/home/thomas/youtube-dl/youtube-dl/youtube_dl/extractor/generic.py", line 2417, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://www.zdf.de/comedy/neo-magazin-mit-jan-boehmermann/neo-magazin-royale-vom-27-10-2016-102.html
...
<end of log>

---

Description of your issue, suggested solution and other information

ZDF just lauched it's new "Mediathek". It was working very fine before, now it doesn't. Some videos can be downloaded on the site (like the one above), but not all. youtube-dl, however, fails with all videos. 😉

[Thor77]

I'll work on this, if noone else started yet.

[bastik]

You beat me to it, here is what I've found so far:

Goal is to download
https://www.zdf.de/service-und-hilfe/die-neue-zdf-mediathek/zdfmediathek-trailer-100.html

1. Get the API token:
   https://www.zdf.de/ZDFplayer/configs/zdf/zdf2016/configuration.json

   => "apiToken": "d272...a32"

2. Get the URL-Template for the meta-data file; use API token as http header like this:

   wget --header='Api-Auth: Bearer d272...a32' \ 'https://api.zdf.de/content/documents/zdfmediathek-trailer-100.json?profile=player'
   => "http://zdf.de/rels/streams/ptmd-template": "/tmd/2/{playerId}/vod/ptmd/mediathek/161021_mt3_supermarkt2_trz"

3. Get the Player-ID:

   https://www.zdf.de/ZDFplayer/latest-v2/skins/zdf/zdf-player.js
   => this.ptmd_player_id="ngplayer_2_3"

4. Insert Player-ID into URL-Template for the meta-data file and download:

   https://api.zdf.de/tmd/2/ngplayer_2_3/vod/ptmd/mediathek/161021_mt3_supermarkt2_trz

This file will contain download URLs etc.

[Thor77]

Wow, that's a great write-up. If you want to, feel free to start implementing that.
I didn't start yet :)

[bastik]

Okay, I'll give it a try.

[isotopp]

#11063 has patches.

[retsifp]

What is the problem to merge #11063, @dstftw? 🙂

[devurandom]

Same problem with https://www.zdf.de/filme/das-kleine-fernsehspiel/die-pruefung-100.html

Is a solution in sight?

[retsifp]

Well, I use the branch from #11063 for weeks now and it works great! No idea why @dstftw doesn't want to merge. 😔

[Hrxn]

Probably because the ZDFChannelIE extractor is not finished yet..
But I agree, if the rest works so far, I think merging it would be better.