ERROR: Unable to extract yt initial data on playlist #32499
Comments
|
this happens on the nightly build: [debug] System config: [] |
|
YT is now ignoring the See also yt-dlp/yt-dlp#7594. |
|
Hello! Do you have instructions on how to pass the cookies from a browser session? Thanks! |
|
See manual and FAQ. |
|
I came accross the issue with consent yesterday. For people using qutebrowser, you first need to convert its cookies into netscape format. I used https://gist.github.com/guidocella/a272b6e68f9c44532b011f6596e95c61#file-dump-cookies-sh to do so and then youtube-dl --cookies $XDG_RUNTIME_DIR/cookies.txt Just posting this in case it saves some people some trouble. |
|
Using a VPN to move from the EU to outside the EU is also a workaround. |
So I updated to latest version with -U, then I passed cookies from my browser after consenting, making sure they are in netscape format. Still doesn't work... Would it help if I provided my cookies txt file? |
|
You have to have consented in the browser and then export the cookies. Although it may not make any difference, also review #30839 to update to a newer version: it will have an earlier date, but I think the ytdl-patched repo is not currently being updated even though builds are being run. |
|
Thank you for your answer, unfortunately it does not solve the issue, I have now updated to the nightly build but I still get the error message. I made sure to export cookies AFTER consenting to youtube cookies. |
|
That's quite surprising. Perhaps YT also requires you to set the UA header to match the browser that acquired the cookies. At any rate, I'm able to get the playlist with a (slightly fixed - hence why it is) WIP version: $ python -m youtube_dl --flat-playlist 'PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU'
[youtube:tab] PL-nQOC8BAGD2Y9TkorVGRBowfO1ILfFKU: Downloading webpage
[download] Downloading playlist: Superwings S1
[youtube:tab] playlist Superwings S1: Downloading 23 videos
[download] Downloading video 1 of 23
[download] Downloading video 2 of 23
[download] Downloading video 3 of 23
[download] Downloading video 4 of 23
[download] Downloading video 5 of 23
[download] Downloading video 6 of 23
[download] Downloading video 7 of 23
[download] Downloading video 8 of 23
[download] Downloading video 9 of 23
[download] Downloading video 10 of 23
[download] Downloading video 11 of 23
[download] Downloading video 12 of 23
[download] Downloading video 13 of 23
[download] Downloading video 14 of 23
[download] Downloading video 15 of 23
[download] Downloading video 16 of 23
[download] Downloading video 17 of 23
[download] Downloading video 18 of 23
[download] Downloading video 19 of 23
[download] Downloading video 20 of 23
[download] Downloading video 21 of 23
[download] Downloading video 22 of 23
[download] Downloading video 23 of 23
[download] Finished downloading playlist: Superwings S1
$ |
|
This is the output after passing my browser's User Agent header to youtube-dl. BTW - I'm using Brave browser. Unfortunately it still does not work. Note that the cookies file is named cookies2.txt it's because I also tried various methods like signing in to google, to no avail. I also tried changing browser, thinking it might be an issue with Brave's agressive privacy behaviour. So here is the output of a fresh Firefox install, with cookies passed after consent and UA header: Interestingly enough it works with another video. I clicked one of the first things that appeared on my feed (https://www.youtube.com/watch?v=FxdbD-N7pHE) and was greeted with a successfull download with the same parameters. There were errors (unable to extract player URL), but the video was still downloaded succesfully. I posted it here since I think it might be more appropriate: ##32538 (comment) |
|
Sadly, I'm out of ideas, except to expedite the QA on the extractor update. |
|
When I made that comment, I found that it didn't work for my French IP (without a VPN) and did for an Australian IP (PIA VPN to Sydney). But now I'm trying again today in France and it kind of works. I see some tracebacks, but it still seems to download a file. So I think those tracebacks are somehow just for warnings. And I suspect they're unrelated. |
|
Based on #32533 (comment), the patch from yt-dlp solves the problem for EU users. PR here RSN |
If it helps I live in Italy. It doesn't seem to be ISP related either. I tested it on my home network, a mobile hotspot and at work, so 3 different ISPs. I don't know why it should change anything, but I thought it was worth a try. BTW I can download single videos from the playlist I mentioned above, albeit at slower speed.
That's some moderately good news... Should we just move to yt-dlp until a fix is released? |
|
using Italy with current |
|
If the The media links can be extracted from the |
|
@1268 Until the patch arrives, you can apply this diff: diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
index 9c419c0..3bf483c 100644
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -260,16 +260,10 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
cookies = self._get_cookies('https://www.youtube.com/')
if cookies.get('__Secure-3PSID'):
return
- consent_id = None
- consent = cookies.get('CONSENT')
- if consent:
- if 'YES' in consent.value:
- return
- consent_id = self._search_regex(
- r'PENDING\+(\d+)', consent.value, 'consent', default=None)
- if not consent_id:
- consent_id = random.randint(100, 999)
- self._set_cookie('.youtube.com', 'CONSENT', 'YES+cb.20210328-17-p0.en+FX+%s' % consent_id)
+ socs = cookies.get('SOCS')
+ if socs and not socs.value.startswith('CAA'): # not consented
+ return
+ self._set_cookie('.youtube.com', 'SOCS', 'CAI', secure=True) # accept all (required for mixes)
def _real_initialize(self):
self._initialize_consent()Then it works fine again: youtube-dl -v --ignore-config IX6McAMbMIU
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', 'IX6McAMbMIU']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.09.01.1
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-162-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f 31 Mar 2020 - glibc 2.31
[debug] exe versions: ffmpeg N-111711-gd295b6b693-Nico-20230808, ffprobe N-111711-gd295b6b693-Nico-20230808, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] IX6McAMbMIU: Downloading webpage
[youtube] IX6McAMbMIU: Downloading player 0a835141
[debug] [youtube] Decrypted nsig eMLFU58o3Ty3bHwmpN => PP6OPqsJ8SF-SQ
[debug] [youtube] Decrypted nsig JGB6co7DtTFZlRKSTI => YxvEq0bReXPeUg
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://rr2---sn-5hnednsz.googlevideo.com/videoplayback?expire=1694039823&ei=r6r4ZKvYGP666dsP4peCWA&ip=<REDACTED>&id=o-AE2Owhk2eLmFpqf1G0FqTEgP1BwezlGU05dRLXbbL2o1&itag=137&aitags=134%2C136%2C137%2C160%2C243&source=youtube&requiressl=yes&mh=Um&mm=31%2C26&mn=sn-5hnednsz%2Csn-4g5e6ns7&ms=au%2Conr&mv=m&mvi=2&pl=14&initcwndbps=2287500&spc=UWF9f3LCXGIQrdCG4aCP0tADEGG8B6rbIbJyHBMF4A&vprv=1&svpuc=1&mime=video%2Fmp4&ns=jfIQASlWGQvPGTF0VTpiDbwP&gir=yes&clen=25280532&dur=401.767&lmt=1693365754427220&mt=1694017749&fvip=3&keepalive=yes&fexp=24007246%2C24362685&beids=24350018&c=WEB&txp=6219224&n=PP6OPqsJ8SF-SQ&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhANPoyNjGdMsqc_doYoBWsy34L_nEI6wn9EbWwn909atVAiBFqVyq-6cllVcGYPN2R6EsVXyoiD0EeU1llgWIYYoq1Q%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgFz9MBbhqbJ_Ev0EWGo2rRJYdQEZ-1Ck4qdyYCWsq9GYCIBj8HKenFbJkrEyhmp4NSj10w-GqcLqrj17bD8Y7CXQQ'
[dashsegments] Total fragments: 3
[download] Destination: HANABI - les règles du jeu !-IX6McAMbMIU.f137.mp4
[download] 100% of 24.11MiB in 00:02
[debug] Invoking downloader on 'https://rr2---sn-5hnednsz.googlevideo.com/videoplayback?expire=1694039823&ei=r6r4ZKvYGP666dsP4peCWA&ip=<REDACTED>&id=o-AE2Owhk2eLmFpqf1G0FqTEgP1BwezlGU05dRLXbbL2o1&itag=140&source=youtube&requiressl=yes&mh=Um&mm=31%2C26&mn=sn-5hnednsz%2Csn-4g5e6ns7&ms=au%2Conr&mv=m&mvi=2&pl=14&initcwndbps=2287500&spc=UWF9f3LCXGIQrdCG4aCP0tADEGG8B6rbIbJyHBMF4A&vprv=1&svpuc=1&mime=audio%2Fmp4&ns=jfIQASlWGQvPGTF0VTpiDbwP&gir=yes&clen=6504156&dur=401.844&lmt=1693365762273497&mt=1694017749&fvip=3&keepalive=yes&fexp=24007246%2C24362685&beids=24350018&c=WEB&txp=6218224&n=PP6OPqsJ8SF-SQ&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAI0HlshLS04cTLtsvCEmR84feV0gb-jUCImXXCz0VjOoAiA5AFpNKO-xA_2-YXUgP0kxsQiqVwxVsaZI8xRAvjvRRQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgFz9MBbhqbJ_Ev0EWGo2rRJYdQEZ-1Ck4qdyYCWsq9GYCIBj8HKenFbJkrEyhmp4NSj10w-GqcLqrj17bD8Y7CXQQ'
[dashsegments] Total fragments: 1
[download] Destination: HANABI - les règles du jeu !-IX6McAMbMIU.f140.m4a
[download] 100% of 6.20MiB in 00:00
[ffmpeg] Merging formats into "HANABI - les règles du jeu !-IX6McAMbMIU.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:HANABI - les règles du jeu !-IX6McAMbMIU.f137.mp4' -i 'file:HANABI - les règles du jeu !-IX6McAMbMIU.f140.m4a' -c copy -map 0:v:0 -map 1:a:0 'file:HANABI - les règles du jeu !-IX6McAMbMIU.temp.mp4'
Deleting original file HANABI - les règles du jeu !-IX6McAMbMIU.f137.mp4 (pass -k to keep)
Deleting original file HANABI - les règles du jeu !-IX6McAMbMIU.f140.m4a (pass -k to keep)
|
|
thanks but I don't actually use YouTube-DL, I use my own tool. I was just trying to help troubleshoot the potential geo block issue. |
|
sorry, I can't reproduce it myself today (2023.09.07 and 2021.12.17), so just forget about it. |
Is there a PR up for this? Also, this affects individual videos, not only playlists. I think the title of this issue should be edited to |
Any minute now. |
|
RE Cookie extraction. Run (() => {
// Paste value of cookie's header obtained from network tab between the quotes,
// or leave it as empty string.
let COOKIE = '' || document.cookie;
copy(
'# Netscape HTTP Cookie File\n' +
COOKIE.split(/; /g)
.map(e => e.replace('=', '\t'))
.map(e => window.location.hostname.replace('www.', '') + '\tFALSE\t/\tTRUE\t0\t' + e)
.join('\n')
)
})()inside your browser console and the Netscape-formatted cookies will be copied to your clipboard. Paste inside a Note: @dirkf Please promote this method anywhere you can. It's by far the safest and easiest way to get the cookies. |
|
We previously identified an apparently reliable browser extension that generates the necessary cookie file, and that's what the online documentation recommends. |
|
I understand that for the online documentation you probably want to stick with recommending something that just works. But since most people don't know how to extract their cookies, youtube-dl documentation currently makes it look like that an extra, special tool is needed to do that. Perhaps if youtube-dl had a Now I wonder if |
|
It could, but see the pinned security issue #32450 for why |
|
I'll keep this open until there's a fixed release. |
Hey I get undefined when running this |
|
Perhaps the poster has this function from SO. Anyhow, s/copy/return/, but generally #32499 (comment) applies. |
undefined what exactly? You should mention this, since virtually everything is undefined inside a browser console except the things that are not. |
This comment was marked as spam.
This comment was marked as spam.
For any new user googling for that. THAT is the correct interim solution. Let's be honest, google does what they can to block downloading.
|
|
The consent issue is fixed in the nightly build which you should be using. |
Where this code should be added? |
In the console of your web browser. |
and others
Checklist
Verbose log
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '-x', 'https://youtube.com/playlist?list=PLA_zjX3swAf5tsw9KdHOGodzcJ_PptL9a']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29
[debug] exe versions: none
[debug] Proxy map: {}
[youtube:tab] PLA_zjX3swAf5tsw9KdHOGodzcJ_PptL9a: Downloading webpage
ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 534, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/youtube.py", line 2841, in _real_extract
data = self._extract_yt_initial_data(item_id, webpage)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/youtube.py", line 299, in _extract_yt_initial_data
self._search_regex(
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Description
WRITE DESCRIPTION HERE
as you can see above the program fails to download even a single video from the playlist
i was using the minimum of arguments in the command and trying the same command with single videos from the playlist produces a different error: "ERROR: Unable to extract uploader id;" this seems to be consistent across the entire playlist
The text was updated successfully, but these errors were encountered: