[ZDF] Playlist extractor finds and tries to download non-video pages

[chrysn]

• I've verified and I assure that I'm running youtube-dl 2018.06.19

• At least skimmed through the README, most notably the FAQ and BUGS sections

• Searched the bugtracker for similar issues including closed ones

• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

• Bug report (encountered problems with youtube-dl)

---

$ /tmp/youtube-dl --flat-playlist https://www.zdf.de/serien/zarah-wilde-jahre -j -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--flat-playlist', u'https://www.zdf.de/serien/zarah-wilde-jahre', u'-j', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.06.19
[debug] Python version 2.7.15 (CPython) - Linux-4.16.0-2-amd64-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 3.4.2-2, ffprobe 3.4.2-2, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/trailer-zarah-wilde-jahre-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/zarah---wilde-jahre-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/zarah---wilde-jahre-102.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/zarah---wilde-jahre-104.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/zarah---wilde-jahre-112.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/zarah---wilde-jahre-114.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/papi-ist-der-beste-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/statements-von-claudia-eisinger-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/zarah-presseartikel-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/verbraucher/volle-kanne/volle-kanne-7-september-2017-mit-svenja-jung-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/interview-kerkow-und-jansen-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/interview-familie-olsen-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/interview-frauenforum-102.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/featurette-kostueme-und-szenebild-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/zarah-featurette-mit-regisseur-richard-huber-100.html", "_type": "url", "ie_key": "ZDF"}
{"url": "https://www.zdf.de/serien/zarah-wilde-jahre/videos/featurette-drehbuchautoren-produzenten-102.html", "_type": "url", "ie_key": "ZDF"}

---

• Playlist: https://www.zdf.de/serien/zarah-wilde-jahre

---

On the given playlist, youtube-dl identifies several downloadable items. The first of those (trailer, wilde-jahre-{100,102,104,112,114}) work fine, but https://www.zdf.de/serien/zarah-wilde-jahre/zarah-presseartikel-100.html is not actually a video but an article on the topic of the playlist (just open it). Attempting to download the complete playlist (in my case by invoking youtube-dl --download-archive .youtube-dl-archive --no-overwrites --all-subs --write-description --write-info-json --write-annotations --write-all-thumbnails --no-progress --output '%(extractor)s/%(id)s %(title)s/%(title)s (%(format)s).%(ext)s' https://www.zdf.de/serien/zarah-wilde-jahre) successfully downloads a few videos and then fails with the same error message as if I tried to download the article directly:

$ /tmp/youtube-dl https://www.zdf.de/serien/zarah-wilde-jahre/zarah-presseartikel-100.html -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.zdf.de/serien/zarah-wilde-jahre/zarah-presseartikel-100.html', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.06.19
[debug] Python version 2.7.15 (CPython) - Linux-4.16.0-2-amd64-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 3.4.2-2, ffprobe 3.4.2-2, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ZDF] zarah-presseartikel-100: Downloading webpage
[ZDF] zarah-presseartikel-100: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError(u'formitaeten',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/tmp/youtube-dl/youtube_dl/extractor/common.py", line 501, in extract
    ie_result = self._real_extract(url)
  File "/tmp/youtube-dl/youtube_dl/extractor/zdf.py", line 233, in _real_extract
    return self._extract_mobile(video_id)
  File "/tmp/youtube-dl/youtube_dl/extractor/zdf.py", line 194, in _extract_mobile
    for f in document['formitaeten']:
KeyError: u'formitaeten'
Traceback (most recent call last):
  File "/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/tmp/youtube-dl/youtube_dl/extractor/common.py", line 514, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'formitaeten',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

---

As a workaround, I'm using the --ignore-errors flag, but that should only ever be a workaround to be carefully applied and not find its way into regular workflows.

Please either make the playlist extraction detect what videos are, or make the download of a single non-video item in a playlist fail gracefully when such a situation is detected.

[mdosch]

I had the same issue but the error also appeared when I didn't try to download the playlist but put all the URLs in a list:

[ZDF] maskenmord-100: Downloading webpage
[ZDF] maskenmord-100: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError(u'formitaeten',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in extract
    ie_result = self._real_extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/zdf.py", line 235, in _real_extract
    return self._extract_mobile(video_id)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/zdf.py", line 196, in _extract_mobile
    for f in document['formitaeten']:
KeyError: u'formitaeten'
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 542, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'formitaeten',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

https://www.zdf.de/serien/professor-t/maskenmord-100.html

[vinzv]

I get the same error for single video pages download:

$ youtube-dl -v https://www.zdf.de/kultur/kulturdoku/bewegte-republik-deutschland-teil-3-100.html
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.zdf.de/kultur/kulturdoku/bewegte-republik-deutschland-teil-3-100.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.09.28
[debug] Python version 3.6.5 (CPython) - Linux-4.12.14-lp151.28.13-default-x86_64-with-glibc2.3.4
[debug] exe versions: ffmpeg 3.4.4, ffprobe 3.4.4
[debug] Proxy map: {}
[debug] Using fake IP 53.17.50.14 (DE) as X-Forwarded-For.
[ZDF] bewegte-republik-deutschland-teil-3-100: Downloading webpage
[ZDF] bewegte-republik-deutschland-teil-3-100: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError('formitaeten',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/zdf.py", line 236, in _real_extract
    return self._extract_mobile(video_id)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/zdf.py", line 197, in _extract_mobile
    for f in document['formitaeten']:
KeyError: 'formitaeten'
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/zdf.py", line 236, in _real_extract
    return self._extract_mobile(video_id)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/zdf.py", line 197, in _extract_mobile
    for f in document['formitaeten']:
KeyError: 'formitaeten'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 543, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('formitaeten',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.