Extractor error on abc.com

[taco82]

Checklist

• I'm reporting a broken site
• I've verified that I'm running yt-dlp version 2022.02.04. (update instructions)
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue
• I've read about sharing account credentials and I'm willing to share it if required

Region

United States

Description

An episode of the Rookie at https://www.abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight fails to download, with an extractor error (KeyError('video'))

Running in Ubuntu 20.04, version 2022.02.04

I do not have an abc account to share credentials to, but one is not needed in the US to view in a browser.

Verbose log

[debug] Command-line config: ['-vU', '--ffmpeg-location', '/usr/bin/ffmpeg', '-N', '5', 'https://www.abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef] (zip)
[debug] Python version 3.8.10 (CPython 64bit) - Linux-5.4.0-99-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Optional libraries: Cryptodome, sqlite
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[debug] [Go] Extracting URL: https://www.abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight
[Go] 13-fight-or-flight: Downloading webpage
[Go] VDKA26139487: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError('video')); please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the "Broken site" issue template properly. Confirm you are on the latest version using -U
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 612, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/go.py", line 215, in _real_extract
    video_data = self._extract_videos(brand, video_id)[0]
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/go.py", line 159, in _extract_videos
    return self._download_json(
KeyError: 'video'
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 612, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/go.py", line 215, in _real_extract
    video_data = self._extract_videos(brand, video_id)[0]
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/go.py", line 159, in _extract_videos
    return self._download_json(
KeyError: 'video'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1381, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1451, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 642, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e, video_id=self.get_temp_id(url))
yt_dlp.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('video')); please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the "Broken site" issue template properly. Confirm you are on the latest version using -U

[taco82]

Also, I've seen some recent extractor bugs indicate trying --no-geo-bypass so I tried that as well, and no improvement.

[dirkf]

From the UK, using --geo-bypass-country US, the attempt to retrieve JSON from http://api.contents.watchabc.go.com/vp2/ws/contents/3000/videos/001/001/-1/-1/-1/VDKA26139487/-1/-1.json succeeds but the JSON has no video, as the error message says:

{'ver': '3000', 'build': '2170', 'device': '9024', 'brand': '13', 'xmlns': 'http://abc.go.com/vp2/ws/xmlns', 'count': '0', 'totalcount': '0'}

Plainly something's changed since this was added in 2016. A little hacking on the value 3000 in the JSON URL shows that 4000 does return the desired result. Also, no need for explicit --geo-bypass...:

$ python3.9 -m yt_dlp -F -v 'https://abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight'
[debug] Command-line config: ['-F', '-v', 'https://abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: 2068a6031
[debug] Python version 3.9.7 (CPython 32bit) - Linux-4.4.0-210-generic-i686-with-glibc2.23
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Optional libraries: sqlite
[debug] Proxy map: {}
[debug] [Go] Extracting URL: https://abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight
[Go] 13-fight-or-flight: Downloading webpage
[Go] VDKA26139487: Downloading JSON metadata
[debug] Using fake IP 6.227.206.164 (US) as X-Forwarded-For
[Go] VDKA26139487: Downloading JSON metadata
[Go] VDKA26139487: Downloading m3u8 information
WARNING: [Go] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the "Broken site" issue template properly. Confirm you are on the latest version using -U
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for VDKA26139487:
ID                                           EXT RESOLUTION FPS │   FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC    ABR MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
ULNK-aac-English_-_Descriptive_Video_Service mp4 audio only     │                  m3u8_n │ audio only        unknown       [EN] English - Descriptive Video Service
MPD                                          mpd unknown        │                  http   │ unknown           unknown
ULNK-214                                     mp4 256x144     15 │ ~ 67.37MiB  214k m3u8_n │ avc1.4d000c  214k mp4a.40.5  0k
ULNK-414                                     mp4 416x234     30 │ ~130.32MiB  414k m3u8_n │ avc1.4d000d  414k mp4a.40.5  0k
ULNK-752                                     mp4 640x360     30 │ ~236.50MiB  752k m3u8_n │ avc1.4d001e  752k mp4a.40.5  0k
ULNK-1705                                    mp4 960x540     30 │ ~535.97MiB 1705k m3u8_n │ avc1.4d001f 1705k mp4a.40.5  0k
ULNK-1326                                    mp4 1280x720    60 │ ~416.88MiB 1326k m3u8_n │ avc1.4d0020 1326k mp4a.40.5  0k
ULNK-1935                                    mp4 1280x720    60 │ ~608.39MiB 1935k m3u8_n │ avc1.640020 1935k mp4a.40.5  0k
ULNK-6199                                    mp4 1280x720    60 │ ~  1.90GiB 6199k m3u8_n │ avc1.640020 6199k mp4a.40.5  0k

Who knows whether any other value of API version works, but the top G hit for the API domain gives this XML containing this fragment (formatted) with what may be a supporting API version:

<application>
  <author>Disney Mobile</author>
  <copyright>@Disney</copyright>
  <email>support@watchdisneyxd.zendesk.com</email>
  <link>http://watchdisneyxd.com/</link>
  <version>
    <latest>4.0.0</latest>
    <required>4.0.0</required>
  </version>
</application>

Same code/problem/fix for yt-dl.

[taco82]

Confirmed the switch from 3000->4000 did fix the issue.

I tested a couple old URLs and they still download fine with 3000, but also work with the version built with 4000.

https://www.abc.com/shows/the-rookie/episode-guide/season-04/12-the-knock
https://www.abc.com/shows/the-wonder-years/episode-guide/season-01/12-im-with-the-band

Not exactly comprehensive, but at least they're not mutually exclusive.

[HobbyistDev]

It's work for me. Here's the log:

(python_proj2) ~\yt-dlp>python -m yt_dlp -v "https://www.abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight"
[debug] Command-line config: ['-v', 'https://www.abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.29 [9d339c41e] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: 67db96e5a
[debug] Python 3.10.4 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.0.1-full_build-www.gyan.dev (setts), ffprobe 5.0.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2022.05.18.1, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] [Go] Extracting URL: https://www.abc.com/shows/the-rookie/episode-guide/season-04/13-fight-or-flight
[Go] 13-fight-or-flight: Downloading webpage
[Go] VDKA26139487: Downloading JSON metadata
[debug] Using fake IP 6.237.31.126 (US) as X-Forwarded-For
[Go] VDKA26139487: Downloading JSON metadata
[Go] VDKA26139487: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] VDKA26139487: Downloading 1 format(s): ULNK-6200
[debug] Invoking hlsnative downloader on "https://content-aaps1-ur-fifa1.uplynk.com/84d676a8810e4e9c8e7cdf5af05daf1b/g.m3u8?exp=1657546880&ct=a&oid=d874124ecca24c88a3c9575e78686acf&eid=8272ba6107614bfd88404f63e42c01fc&expand=drmOff&iph=a7acbd023d89e015f136520442fe959760436eec08ffe67b5d1217f19401c62a&euid=7152C042-7ACB-43A1-BBEA-DB126E692255_000_0_001_lf_01-06-00_NA&cdn=ec&stgcfg=datg&rays=hjigfedcba&pp2ip=0&v=3&sig=3be019db62bb126d1db6cdbe85809bb70666b920872c23d26b72c748dd157018&pbs=5c464cbc51c8430993001e7f42bbf64c"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 631
[download] Destination: Fight or Flight [VDKA26139487].mp4
[debug] File locking is not supported. Proceeding without locking
[download]   0.8% of ~1.89GiB at    673.15B/s ETA 43:33 (frag 5/631)
ERROR: Interrupted by user

[pukkandan]

Likely because the URL is now "old"