Intermittent "Incomplete data received" errors when downloading YouTube playlist JSON #8206
Comments
stuartlangridge
added
site-bug
|
I can intermittently reproduce the errors. Today was the first I've noticed them |
|
Seeing same here on latest release |
|
Note: I am experiencing this on 2023.07.06 as well; not a regression |
|
I can also reproduce this error. I last tested with two playlists. What I have observed is that yt-dlp claims that when the error occurs there is one more video in the playlist than there really is. Without error, 23 videos in playlist, correct number
With error, supposedly 24 videos in playlist, wrong number
|
|
To add to my comment, I tested with version 2023.07.06 and 2023.09.24 |
|
I've also been experiencing this error with playlists as of the last day or so on 2023.07.06 |
I don't have time to investigate this rn, but some things to check if anyone wants to: What's the 24th video in the playlist that yt-dlp extracts? is it the same as the first? What is the behaviour in the browser with same cookies that is used in yt-dlp that triggers this (i.e same visitor data cookie)?.
|
|
For anyone hitting this error, are you running over a VPN? I noticed that running 2023.09.24 on VPN produces the errors but running same version through local ISP succeeds. |
|
I'm getting this error too. The JSON key If the error happens when downloading the playlist, yt-dlp says |
No, I'm running my commands through my local isp when observing the issue. |
|
I'm using the yt-dlp Python package v2023.9.24 in my python video downloader script, and I am getting this error "ERROR: Incomplete data received" on what appears to be the last entry of certain playlists. I haven't been able to isolate it what is causing it, but I added a try/catch block around it to ignore the exception, so that it at least runs. When it fails, it fails on " for entry in result['entries']:" line so there is some issue with item in the list. And example list is https://www.youtube.com/playlist?list=PLR7yrLMHm11X6-M_usCj5H-gdstyWNLXQ . I stepped though the "for url in url_list" loop when it got to the last entry in the playlist, it threw the exception. |
|
Same, affects MPV too: Hopefully it will be fixed soon. |
coletdjnz
added
needs-investigating
happens for me both with and without proxy servers also, this playlist reliably fails with "Incomplete data received" every time yt-dlp "https://www.youtube.com/playlist?list=PLdJo8g6QW5jbOPhBUj5ferM3FDkmmTHnp" --flat-playlist -J |
|
Note that this issue is non-fatal, so everything should still work as expected by default. If you are embedding yt-dlp, set |
In MPV case, setting Test case: yt-dlp's command line: I tried to send all kind of error suppressings but the only success was to modify |
|
Doing some digging, this appears to be a bug on YouTube's end, but we can take some steps in yt-dlp to handle it better. In YouTube, if you own the playlist, you get a "recommended videos shelf" at the end of the playlist. It seems like yt-dlp tries to request this page (which is expected given the structure), but doesn't extract any videos (the traverser doesn't support the path required) However, it seems like in a recent YouTube update they are adding this continuation for the recommended videos shelf to any playlist - even if you don't own it. This seems to be a bug on their end since it never returns with any videos/data in this case. You can see this for yourself in the browser - when you load a 1 page playlist that is not your own, there is a divider at the end and there is a spinner for a moment as it tries loads this shelf that has no videos/data.
What I'm thinking of doing:
|
More details over at yt-dlp: yt-dlp/yt-dlp#8206
|
Fails for me as well with the very first request: import yt_dlp
ydl_opts: dict = {
'no_warnings': True,
'quiet': True,
}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
ydl.extract_info('https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS', download=False)
ERROR: Incomplete data received
Traceback (most recent call last):
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1567, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 2427, in <lambda>
return type(self.ydl)._handle_extraction_exceptions(lambda _, i: self._entries[i])(self.ydl, i)
~~~~~~~~~~~~~^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 2195, in __getitem__
self._cache.extend(itertools.islice(self._iterable, n))
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/extractor/youtube.py", line 4933, in _entries
response = self._extract_response(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/extractor/youtube.py", line 944, in _extract_response
for retry in self.RetryManager():
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 5112, in __iter__
self.error_callback(self.error, self.attempt, self.retries)
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 3760, in _error_or_warning
RetryManager.report_retry(
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 5120, in report_retry
raise e
yt_dlp.utils.ExtractorError: Incomplete data received
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1556, in extract_info
return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1567, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1723, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1852, in process_ie_result
return self.__process_playlist(ie_result, download)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1927, in __process_playlist
entries = resolved_entries = list(entries)
^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 738, in _iter
for x in iterable:
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 2392, in get_requested_items
for i, entry in self[index]:
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 2454, in __getitem__
entry = self._getter(i)
^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 2427, in get_entry
return type(self.ydl)._handle_extraction_exceptions(lambda _, i: self._entries[i])(self.ydl, i)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1585, in wrapper
self.report_error(str(e), e.format_traceback())
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1045, in report_error
self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
File "/home/sebastian/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 984, in trouble
raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: Incomplete data receivedthis happens for all of the following playlists: playlists = [
'https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS',
'https://www.youtube.com/playlist?list=PL-osiE80TeTvipOqomVEeZ1HRrcEvtZB_',
'https://www.youtube.com/playlist?list=PL5-da3qGB5IBITZj_dYSFqnd_15JgqwA6',
'https://www.youtube.com/playlist?list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y',
'https://www.youtube.com/playlist?list=PLB4tvLCynFjSixzoz4z2BGDCWxSW6rCRU',
"https://www.youtube.com/playlist?list=PLB4tvLCynFjSafzzFD9AP0FrtAXAyMx0m",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjQGLrVOAzzwHYrCDyrZ03H1",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjRmGIVW4PuYzdzqH9vk91eR",
'https://www.youtube.com/playlist?list=PLB4tvLCynFjQjcVftzhNm4As8Dn69PKa1',
'https://www.youtube.com/playlist?list=PLB4tvLCynFjRQuv9-z0xYe3zoMmbL3sS-',
"https://www.youtube.com/playlist?list=PLB4tvLCynFjTTSXUmzM_9WVJiVGoX3axg",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjRvNKAErLi7uduuVTdzbtvd",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjSzpDc0dvoA_DVRYybrQReu",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjRJchIJjKiGGhW4rI5kbmql",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjQpENhUwb-WGNyYUo9F1r1B",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjQm7ikab5mM4usj3-n3Bf7I",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjQj4F_rTGoZkX_eF2yovMz9",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjQSNMgYAdbBokLEPpgPh6iF",
"https://www.youtube.com/playlist?list=PLB4tvLCynFjSXxsJ-8_2fgej53MyzAKFb",
]No issues at all when extracting information from channel URLs though. |
|
Can confirm, works with the commit from 3 days ago. |
|
Actually, not quite: It still loops through the "Incomplete data received" warning, wasting time and bandwidth, and with |
Don't use The bug is on YouTube's end and there have been other similar cases where retries don't help. Just that it's not always possible to tell the difference between when they can and not. |
|
It appears to be fixed. I removed my workaround and ran yt-dlp-2023.10.7 Python module both in an outside of a debugger without issue. Thank you for the fix! |
|
I'm on 2023.11.16 and I get this problem on every Playlist I try to download. |
|
I used it on a playlist 11 hours ago and it worked. Please create a new bug report with verbose output enabled. |
|
yt-dlp#8238) Closes yt-dlp#8206 Adds `raise_incomplete_data` extractor arg to revert this behaviour and raise an error. Authored by: coletdjnz Co-authored-by: Simon Sawicki <contact@grub4k.xyz>
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
Checklist
Region
United Kingdom
Provide a description that is worded well enough to be understood
When running the following command:
yt-dlp -vU --dump-single-json --flat-playlist PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFfit sometimes succeeds, and sometimes fails with
yt_dlp.utils.ExtractorError: Incomplete data receivedas can be seen below. It does not always fail; this error seems to be intermittent. It is also not specific to this playlist; I have seen it fail with the same error on other playlists as well. (The specific playlist mentioned above is https://www.youtube.com/playlist?list=PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf for convenience of linking; it opens in a browser successfully.) In running this command 5 times, it succeeded twice and failed three times.Provide verbose output that clearly demonstrates the problem
yt-dlp -vU <your command line>)'verbose': TruetoYoutubeDLparams instead[debug] Command-line config) and insert it belowComplete Verbose Output
The text was updated successfully, but these errors were encountered: