Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All videos broken #486

Closed
cloudrac3r opened this issue Sep 11, 2019 · 26 comments
Closed

All videos broken #486

cloudrac3r opened this issue Sep 11, 2019 · 26 comments

Comments

@cloudrac3r
Copy link

Seems like YouTube has made a breaking change as of a few minutes ago. Every video ID that I've tried is broken in ytdl-core as well as youtube-dl and Lavaplayer. Interestingly, Invidious seems to have survived this change.

youtube-dl issue: ytdl-org/youtube-dl#22367

> ytdl = require("ytdl-core")
...
> ytdl.getInfo("kJQP7kiw5Fk")
Promise { <pending> }
> (node:389040) UnhandledPromiseRejectionWarning: Error: This video is unavailable
	 at /tmp/tmp.h8A12K3WoR/node_modules/ytdl-core/lib/info.js:255:16
	 at /tmp/tmp.h8A12K3WoR/node_modules/ytdl-core/lib/info.js:373:9
	 at /tmp/tmp.h8A12K3WoR/node_modules/ytdl-core/lib/info.js:198:5
	 at PassThrough.<anonymous> (/tmp/tmp.h8A12K3WoR/node_modules/miniget/lib/index.js:195:30)
	 at PassThrough.emit (events.js:209:13)
	 at PassThrough.EventEmitter.emit (domain.js:499:23)
	 at endReadableNT (_stream_readable.js:1178:12)
	 at processTicksAndRejections (internal/process/task_queues.js:80:21)
@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

A few reports of the same error from a few minutes ago are also in the thread for issue #371.

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

It seems like fmt_list is not longer included in the server info response.

@WaqasIbrahim
Copy link
Contributor

WaqasIbrahim commented Sep 11, 2019

Formats and adaptive formats are also available in palyerResponse. We can use that?

const player_response = config.args.player_response || info.player_response;

Not sure how reliable this data is, I mean does it always have the formats.

@AuKllM
Copy link

AuKllM commented Sep 11, 2019

I have the same problem..

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

According to the test data files, player_response.streamingData used to just contain a property called "expiresInSeconds". Now it contains much more information:

{
    "expiresInSeconds": "21540",
    "formats": [
        {
            "itag": 18,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=18&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=22828020&ratebypass=yes&dur=262.199&lmt=1392560250516306&mt=1568214966&fvip=3&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&sig=ALgxI2wwRgIhAIBQdl7vu9ANBBYfG-XtBlzC2I815baeEa1GoSpV5LaCAiEAgMYaWZqOd9p9qajYkqAKJgtVhF_IaYwEn0YhzllL-18%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.42001E, mp4a.40.2\"",
            "bitrate": 696698,
            "width": 480,
            "height": 360,
            "lastModified": "1392560250516306",
            "contentLength": "22828020",
            "quality": "medium",
            "qualityLabel": "360p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 696509,
            "audioQuality": "AUDIO_QUALITY_LOW",
            "approxDurationMs": "262199",
            "audioSampleRate": "44100",
            "audioChannels": 1
        }
    ],
    "adaptiveFormats": [
        {
            "itag": 134,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=134&aitags=133%2C134%2C160&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=15664974&dur=262.096&lmt=1392560231114616&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRAIgAlKav-dUE-CZBsB0SLAkNZu8ml_ZP3lWHOg_s0iTVHMCIAPAC8W1Tpm9-NiR5EppFhtoDfrLduqtYDmSsvmc9zNi&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.4d401e\"",
            "bitrate": 606491,
            "width": 480,
            "height": 360,
            "initRange": {
                "start": "0",
                "end": "708"
            },
            "indexRange": {
                "start": "709",
                "end": "1376"
            },
            "lastModified": "1392560231114616",
            "contentLength": "15664974",
            "quality": "medium",
            "fps": 30,
            "qualityLabel": "360p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 478144,
            "highReplication": true,
            "approxDurationMs": "262096"
        },
        {
            "itag": 133,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=133&aitags=133%2C134%2C160&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=8000185&dur=262.096&lmt=1392560291567194&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRAIgH07C9HGn7PDiq_g4c8qK48Rcm2jhanCmKo0KlTOSPM0CIFjFiY95ED3elszkagrnNtwXRmiNQ-Via6XbOgqiO1JR&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.4d400d\"",
            "bitrate": 251222,
            "width": 320,
            "height": 240,
            "initRange": {
                "start": "0",
                "end": "671"
            },
            "indexRange": {
                "start": "672",
                "end": "1339"
            },
            "lastModified": "1392560291567194",
            "contentLength": "8000185",
            "quality": "small",
            "fps": 30,
            "qualityLabel": "240p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 244190,
            "approxDurationMs": "262096"
        },
        {
            "itag": 160,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=160&aitags=133%2C134%2C160&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=3698377&dur=262.096&lmt=1392560205510693&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRQIgftzDulcXjVq-xAn0j26hUY4Gmz0eojnNg08YvcXpcpoCIQCsTAt4QT_B2MTbotgdaxQw-xj-vd1zVwJhiIFR41Rt_w%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.4d400c\"",
            "bitrate": 118901,
            "width": 192,
            "height": 144,
            "initRange": {
                "start": "0",
                "end": "672"
            },
            "indexRange": {
                "start": "673",
                "end": "1340"
            },
            "lastModified": "1392560205510693",
            "contentLength": "3698377",
            "quality": "tiny",
            "fps": 15,
            "qualityLabel": "144p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 112886,
            "approxDurationMs": "262096"
        },
        {
            "itag": 140,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=140&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=audio%2Fmp4&gir=yes&clen=3116224&dur=262.199&lmt=1392560200450967&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRQIgDxUMLNu4aRY6MxWAlB8APnhWQdL0Ib3-bEKfv9s1z3UCIQD1x9awmo6QoqyBNQduRCyWI1IhjP_1wOvV8tGQiIBcLA%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "audio/mp4; codecs=\"mp4a.40.2\"",
            "bitrate": 95952,
            "initRange": {
                "start": "0",
                "end": "591"
            },
            "indexRange": {
                "start": "592",
                "end": "947"
            },
            "lastModified": "1392560200450967",
            "contentLength": "3116224",
            "quality": "tiny",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 95079,
            "highReplication": true,
            "audioQuality": "AUDIO_QUALITY_MEDIUM",
            "approxDurationMs": "262199",
            "audioSampleRate": "44100",
            "audioChannels": 1
        },
        {
            "itag": 251,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=251&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=audio%2Fwebm&gir=yes&clen=3303678&dur=262.221&lmt=1565114931492150&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRQIgIOMc191XuYazkHIn90mmKg40ZuNm4t9PTX18ctLZv0cCIQD3wncti_OfYsKEnCxJVZitu4isQC4BBG15vntQOGJ7uA%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "audio/webm; codecs=\"opus\"",
            "bitrate": 115903,
            "initRange": {
                "start": "0",
                "end": "258"
            },
            "indexRange": {
                "start": "259",
                "end": "714"
            },
            "lastModified": "1565114931492150",
            "contentLength": "3303678",
            "quality": "tiny",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 100790,
            "audioQuality": "AUDIO_QUALITY_MEDIUM",
            "approxDurationMs": "262221",
            "audioSampleRate": "48000",
            "audioChannels": 1
        }
    ]
}

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

At least for the _HSylqgVYQI regular test case, player_response.streamingData.formats contains the same formats that were previously in fmt_list.

@omarroth
Copy link

Yep, should be possible to pull all necessary info from streamingData.formats and streamingData.adaptiveFormats. For reference.

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

The signature stuff is now included in each player_response.streamingData.adaptiveFormat's cipher field. I hope the s param is the same format as before.

@WaqasIbrahim
Copy link
Contributor

@FD- Thanks, I was just about to ask. I can't find a video with signature data. Can you share a link?

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

@WaqasIbrahim It's from the vevo test case: https://www.youtube.com/watch?v=pJk0p-98Xzc

@a26364177
Copy link

temp fix like this
ytdl-org/youtube-dl@bf1317d

@WaqasIbrahim
Copy link
Contributor

@FD- Thanks again. Looks like cipher is a query string that has the url, sp and s. And signature deciphering is working fine with my testing.

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

Working fine for me as well. Actually, a quick fix is implemented in just a few lines in parseFormats (but I'm using a modified version of the library, so might be different for unmodified). All in all, a few fields were moved to different places and some were renamed to camel casing.

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

What I still don't know though:

  • Where are dash manifests now?
  • Are subtitles still in info.player_response.captions? Yes, just as before.
  • Is the request to get_info still necessary now that we already get the streams in the config? Likely no longer needed

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

Indeed seems like the request to get_info can be removed (sorry, had a mistake in an update to my comment above). The only remaining reason seemed to be to check whether requires_purchase is set, but that property seems no longer working either. Instead, videos that require a purchase can be detected via player_response.playabilityStatus:

{"status":"UNPLAYABLE","reason":"This video requires payment to watch.","errorScreen":...}

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

A quick and dirty fix is to add this to the end of the parseFormats function:

formats = formats.concat(info.player_response.streamingData.formats);
formats = formats.concat(info.player_response.streamingData.adaptiveFormats);
formats.forEach(function(format){
    if (format.cipher){
        format.cipher = parseQueryString(format.cipher);
        Object.assign(format, format.cipher);
    }
    format.type = format.mimeType;
    format.quality_label = format.qualityLabel;
});
info.length_seconds = info.player_response.videoDetails.lengthSeconds;

Please note that this is in no way a permanent solution. The permanent fix likely requires modifications to quite a few places within the library.

@WaqasIbrahim
Copy link
Contributor

WaqasIbrahim commented Sep 11, 2019

    format.cipher = parseQueryString(format.cipher);

It should be querystring.parse(format.cipher);

Also formats have bitrate now (number), so we get an error here.

let s = c.bitrate.split('-');

And itags are also numbers in formats so chooseFormat does not work with itags.

return formats.find((format) => format.itag === '' + itag);

Quick fix:

formats = formats.concat(info.player_response.streamingData.formats);
formats = formats.concat(info.player_response.streamingData.adaptiveFormats);
formats.forEach(function(format){
    if (format.cipher){
        format.cipher = querystring.parse(format.cipher);
        Object.assign(format, format.cipher);
    }
    format.type = format.mimeType;
    format.quality_label = format.qualityLabel;
    format.itag = format.itag.toString();
    delete format.bitrate;
});
info.length_seconds = info.player_response.videoDetails.lengthSeconds;

@JosefBud
Copy link

@WaqasIbrahim this is the one that worked for me (for a Discord bot). Thank you!

@FD-
Copy link
Contributor

FD- commented Sep 11, 2019

@WaqasIbrahim Thanks for the improvements! As mentioned, I‘m using a modified version of node-ytdl-core, and didn’t test with the public version.

@Roki100
Copy link
Contributor

Roki100 commented Sep 11, 2019

@FD- wdym modified version?

@fent
Copy link
Owner

fent commented Sep 12, 2019

is this still an issue? just tested this on a few random videos and not getting any errors

@fent
Copy link
Owner

fent commented Sep 12, 2019

Indeed seems like the request to get_info can be removed (sorry, had a mistake in an update to my comment above). The only remaining reason seemed to be to check whether requires_purchase is set, but that property seems no longer working either. Instead, videos that require a purchase can be detected via player_response.playabilityStatus:

yeah I think it's safe to remove, most of what it was used for can be found in player_response.videoDetails

@FD-
Copy link
Contributor

FD- commented Sep 12, 2019

@FD- wdym modified version?

I'm using it in a different custom runtime (not node), so had to remove the dependencies and rewrite some parts.

is this still an issue? just tested this on a few random videos and not getting any errors

Wow, adaptive_fmts and fmt_list are back again! Seems like player_response.streamingData still contains formats and adaptiveFormats though. No idea whether they are in any way better to use than adaptive_fmts and fmt_list.
Also seems like info.length_seconds is still gone.

@WaqasIbrahim
Copy link
Contributor

@FD- wdym modified version?

I'm using it in a different custom runtime (not node), so had to remove the dependencies and rewrite some parts.

is this still an issue? just tested this on a few random videos and not getting any errors

Wow, adaptive_fmts and fmt_list are back again! Seems like player_response.streamingData still contains formats and adaptiveFormats though. No idea whether they are in any way better to use than adaptive_fmts and fmt_list.
Also seems like info.length_seconds is still gone.

streamingData is already parsed and contains content length and other useful info.

@SergioGlorias
Copy link

I'm no longer having problems with ytdl

yesterday yes I was in trouble

but not today

@developmedev
Copy link

var videoPageContent = client.GetStringAsync(videoInfoUrl).Result;
var videoParameters = HttpUtility.ParseQueryString(videoPageContent);
var encodedStreamsDelimited1 = WebUtility.HtmlDecode(videoParameters["player_response"]);
JObject jObject = JObject.Parse(encodedStreamsDelimited1);
string url = (string) jObject["streamingData"]["formats"][0]["url"];
return VideoSource.FromUri(url);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests