Add parameters to `get_video_info` endpoint to fix 404 errors #938

D3SOX · 2021-06-19T22:41:52Z

This fixes 404 errors.
See zerodytrash/Simple-YouTube-Age-Restriction-Bypass#18 (comment) for more information about this issue

Fixes #939

fixes 404 errors

BenCarmichael01 · 2021-06-22T14:02:35Z

this fixes the 404 error but it also seems to add about 5 second delay to a getInfo as well

TimeForANinja · 2021-06-22T14:13:33Z

lib/info.js

@@ -325,6 +325,8 @@ const VIDEO_EURL = 'https://youtube.googleapis.com/v/';
 const getVideoInfoPage = async(id, options) => {
  const url = new URL(`https://${INFO_HOST}${INFO_PATH}`);
  url.searchParams.set('video_id', id);
+  url.searchParams.set('c', 'TVHTML5');
+  url.searchParams.set('cver', '7.20190319');


is it possible to get this number from one of the prev. fetched pages? not a big fan of hard coding version numbers... 😅

Not sure, I got the information from @zerodytrash. StackOverflow answers also suggest 6.20180913 for the version
https://stackoverflow.com/questions/67615278/get-video-info-youtube-endpoint-suddenly-returning-404-not-found

I tried it with other random numbers and strings and it seems to work too so I guess it only has to be set to something.

actually just searching the watch page gave me this

cver: 7.20190319 is over 1 year old...

Should I change it to that value?

setting cver to 2.20210621.02.00 seems to cause the issue all over again

ok seems like to set cver to 2.20210621.02.00,you need to set c to WEB. but when i did this it still caused 404s sometimes

for now i am applying a local patch with c set to TVHTML5 and cver set to 6.20180913

D3SOX · 2021-06-22T14:22:18Z

seems to add about 5 second delay

@Jaques22 Hm I didn't notice any delay. Are you sure this is the culprit?

D3SOX · 2021-06-22T14:23:44Z

@AlbinGrans Are those defined when you send the request without those parameters? Or is it just giving a 404 instead?
Maybe they renamed some parameters. I haven't looked at those yet.

BenCarmichael01 · 2021-06-22T14:50:43Z

seems to add about 5 second delay

@Jaques22 Hm I didn't notice any delay. Are you sure this is the culprit?

I should note that I am using ytdl-core-discord which relies upon this so maybe a problem with that wrapper.
I just did some very basic testing and it seems like the getinfo call takes longer now

try { 
                const results = await youtube.searchVideos(search, 1, { part: "snippet" });
                console.log(results);
                console.log("start getInfo");
                songInfo = await ytdl.getInfo(results[0].url);
                console.log("end getInfo");
...

results definitely returns before the "start" log then it's about 5 seconds before the "end" log whereas before it was maybe 1-2s.
Though with this being a youtube api change its probably on the api side not ytdl

decahedron1 · 2021-06-22T15:13:20Z

New endpoint removes microformat metadata as well

TimeForANinja · 2021-06-22T15:51:25Z

results definitely returns before the "start" log then it's about 5 seconds before the "end" log whereas before it was maybe 1-2s.
Though with this being a youtube api change its probably on the api side not ytdl

have heard of that befor in another context 🤔 so this might not be related to the 404's

BenCarmichael01 · 2021-06-22T16:02:27Z

results definitely returns before the "start" log then it's about 5 seconds before the "end" log whereas before it was maybe 1-2s.
Though with this being a youtube api change its probably on the api side not ytdl

have heard of that befor in another context 🤔 so this might not be related to the 404's

Yeh it's probably not related, just thought I'd add in case a faster alternative was doable but I guess we'll see if someone comes up with one

D3SOX · 2021-06-22T23:40:20Z

The metadata seems to have changed with this. The duration no longer works. I might investigate this later today. If anyone has an idea let me know

JeHwanYoo · 2021-06-23T06:10:38Z

it perfectly works thank you.
I'll use this temporarily until the official version is updated.

Zigecek · 2021-06-23T10:31:46Z

Hi, when will be this merged?

D3SOX · 2021-06-23T12:15:55Z

@timbly5000 I personally use patch-package for now. You can apply the patch inside node_modules and then run patch-package ytdl-core or put this into a file named ytdl-core+4.8.2.patch in a patches folder relative the root of your project

diff --git a/node_modules/ytdl-core/lib/info-extras.js b/node_modules/ytdl-core/lib/info-extras.js
index c806c9f..0cfe685 100644
--- a/node_modules/ytdl-core/lib/info-extras.js
+++ b/node_modules/ytdl-core/lib/info-extras.js
@@ -283,7 +283,7 @@ exports.cleanVideoDetails = (videoDetails, info) => {
   // Use more reliable `lengthSeconds` from `playerMicroformatRenderer`.
   videoDetails.lengthSeconds =
     info.player_response.microformat &&
-    info.player_response.microformat.playerMicroformatRenderer.lengthSeconds;
+    info.player_response.microformat.playerMicroformatRenderer.lengthSeconds || info.player_response.videoDetails.lengthSeconds;
   return videoDetails;
 };
 
diff --git a/node_modules/ytdl-core/lib/info.js b/node_modules/ytdl-core/lib/info.js
index 417f876..a2dd09a 100644
--- a/node_modules/ytdl-core/lib/info.js
+++ b/node_modules/ytdl-core/lib/info.js
@@ -325,6 +325,8 @@ const VIDEO_EURL = 'https://youtube.googleapis.com/v/';
 const getVideoInfoPage = async(id, options) => {
   const url = new URL(`https://${INFO_HOST}${INFO_PATH}`);
   url.searchParams.set('video_id', id);
+  url.searchParams.set('c', 'TVHTML5');
+  url.searchParams.set('cver', '7.20190319');
   url.searchParams.set('eurl', VIDEO_EURL + id);
   url.searchParams.set('ps', 'default');
   url.searchParams.set('gl', 'US');

D3SOX · 2021-06-23T12:43:57Z

Only player_response.videoDetails.lengthSeconds seems to be set with these parameters. I added it as a fallback to be included in the info result's videoDetails property

@AlbinGrans Can you confirm that it's working now?

AlbinGrans · 2021-06-23T14:25:03Z

Only player_response.videoDetails.lengthSeconds seems to be set with these parameters. I added it as a fallback to be included in the info result's videoDetails property

@AlbinGrans Can you confirm that it's working now?

Yep lengthSeconds is there now! publishDate is still undefined.

1001Josias

This change worked perfectly for me.

TimeForANinja · 2021-06-23T23:08:47Z

ok - thinking about giving this a pass, even though this breaks basically all videoDetails
it's the fix youtube-dl went for and i don't see any other option
did test some with other cver but no success... hope this doesn't end in a "update cver every 3 months"

skick1234 · 2021-06-24T01:04:54Z

did test some with other cver but no success... hope this doesn't end in a "update cver every 3 months"

We can get the client version from the web and replace 2.version with 7.version to change it to TV client version.

Like 2.20210622.10.00 -> 7.20210622.10.00

mpirescarvalho · 2021-06-24T01:29:45Z

It's not fully working for me, still getting 404 ~~in, let's say, half of the requests~~

Edit: yeah, definitively not working for me

TimeForANinja · 2021-06-24T01:45:14Z

did test some with other cver but no success... hope this doesn't end in a "update cver every 3 months"

We can get the client version from the web and replace 2.version with 7.version to change it to TV client version.

Like 2.20210622.10.00 -> 7.20210622.10.00

that sound's promising... didn't try that yet

D3SOX · 2021-06-24T13:13:44Z

did test some with other cver but no success... hope this doesn't end in a "update cver every 3 months"

We can get the client version from the web and replace 2.version with 7.version to change it to TV client version.

Like 2.20210622.10.00 -> 7.20210622.10.00

Are you sure the version numbers work like that? If so, we would need to pass the information from getWatchHTMLPage.

I did a little test (with #946 applied) and adding this at the bottom of getWatchHTMLPage seems to do it:

const clientVersion = info.response.responseContext.serviceTrackingParams
    .find(param => param.service === 'CSI').params
    .find(param => param.key === 'cver').value;
const tvVersion = `7${clientVersion.substring(1)}`;

(quick and dirty without any sanity checks)

If this is going to be the way to go, how do we want to implement it? I thought about setting it to something like options.tvVersion and then using it inside getVideoInfoPage.

What should we use as a fallback value if getWatchHTMLPage fails?

TimeForANinja · 2021-06-24T16:05:40Z

thought about using utils.between(body, '{"key":"cver","value":"', '"}')
that would make it independant of finding a player config

besides that, i don't think that

What should we use as a fallback value if getWatchHTMLPage fails?

is a real problem. If getWatchHTMLPage completely fails something is wrong with the video. If we don't find a player config we should still find the cver

another option would be just caching it in a global variable... default value could be sth. like 7.20210622.10.00, update it every time we do any getWatchHTMLPage and it would therefor always be present

jbelford · 2021-06-24T22:42:07Z

Thought I'd share this PR on lavaplayer sedmelluq/lavaplayer#640

Tested on my own branch and c=ANDROID and cver=16.24 also works

mome0320 · 2021-06-25T01:23:50Z

Thought I'd share this PR on lavaplayer sedmelluq/lavaplayer#640

Tested on my own branch and c=ANDROID and cver=16.24 also works

It also lengthSeconds, publishDate is still undefined.

D3SOX · 2021-06-25T09:43:49Z

@TimeForANinja Thank you, that should do it.

Lazy-C0der

at line 312 in info.js can you explain me what actually this code do ??

omenocal · 2021-06-25T17:10:39Z

@TimeForANinja thanks for looking into this issue. I've got a large project that heavily relies on this library, do you have an ETA when a new release with these fixes will be published to NPM? Thanks,

TimeForANinja · 2021-06-25T17:29:02Z

@TimeForANinja thanks for looking into this issue. I've got a large project that heavily relies on this library, do you have an ETA when a new release with these fixes will be published to NPM? Thanks,

Waiting for some feedback from the other devs
Might merge after that

cherry-picked from distubejs@e13c2a9

TimeForANinja · 2021-06-26T03:27:13Z

Thanks again for the PR @D3SOX and everyone else who helped
For further Problems with (especially) video details check out #958

github-actions · 2021-06-26T03:27:19Z

🎉 This PR is included in version 4.8.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

fix: add parameters to get_video_info endpoint

d1f5357

fixes 404 errors

TimeForANinja reviewed Jun 22, 2021

View reviewed changes

D3SOX changed the title ~~Add useful parameters to get_video_info endpoint~~ Add parameters to get_video_info endpoint to fix 404 errors Jun 22, 2021

D3SOX added 2 commits June 23, 2021 14:36

fix: add fallback for lengthSeconds

da8f371

fix: correct lengthSeconds path

ffb16b8

D3SOX requested a review from TimeForANinja June 23, 2021 12:45

efb4f5ff-1298-471a-8973-3d47447115dc mentioned this pull request Jun 23, 2021

Can't load certain videos FreeTubeApp/FreeTube#1438

Closed

1001Josias approved these changes Jun 23, 2021

View reviewed changes

yalanyali approved these changes Jun 23, 2021

View reviewed changes

style: fix linter errors

faab2d6

TimeForANinja mentioned this pull request Jun 23, 2021

Is not working any more #945

Closed

TimeForANinja mentioned this pull request Jun 24, 2021

Update getWatchHTMLPage right pos #946

Merged

200percentmicky added a commit to 200percentmicky/chadtube that referenced this pull request Jun 24, 2021

See fent/node-ytdl-core#938

4f68bf8

skick1234 mentioned this pull request Jun 24, 2021

This fixes my issues. #948

Closed

CALO77103 mentioned this pull request Jun 24, 2021

[Bug Report] Error 404 stegripe/rawon#333

Closed

redbrain mentioned this pull request Jun 25, 2021

Update info.js #951

Closed

Sleepytt1210 mentioned this pull request Jun 25, 2021

lengthSeconds undefined #949

Closed

Update info.js

deb7410

Lazy-C0der reviewed Jun 25, 2021

View reviewed changes

TimeForANinja mentioned this pull request Jun 25, 2021

Does not work any more #954

Closed

bees mentioned this pull request Jun 25, 2021

add patch for ytdl error vphoebe/beatnik#21

Merged

refactor: use cver for x-youtube-client-version header

1ec2bc5

cherry-picked from distubejs@e13c2a9

muhitrhn mentioned this pull request Jun 26, 2021

Could not get Error: Status Code: 429 fixed. #957

Closed

TimeForANinja merged commit 977e973 into fent:master Jun 26, 2021

TimeForANinja mentioned this pull request Jun 26, 2021

Inconsistent VideoDetails #958

Open

github-actions bot added the released label Jun 26, 2021

TimeForANinja mentioned this pull request Jun 26, 2021

MinigetError: input stream: Status code: 404 #939

Closed

D3SOX deleted the fix-info-endpoint-404 branch June 26, 2021 12:23

th-ch mentioned this pull request Jun 26, 2021

Downloader Stuck On Downloading th-ch/youtube-music#338

Closed

phyzical mentioned this pull request Jun 27, 2021

Error ! in downloading 404 code ! SwapnilSoni1999/spotify-dl#109

Closed

Add parameters to get_video_info endpoint to fix 404 errors #938

Add parameters to get_video_info endpoint to fix 404 errors #938

Conversation

D3SOX commented Jun 19, 2021 • edited

BenCarmichael01 commented Jun 22, 2021

TimeForANinja Jun 22, 2021

Choose a reason for hiding this comment

D3SOX Jun 22, 2021

Choose a reason for hiding this comment

TimeForANinja Jun 22, 2021

Choose a reason for hiding this comment

D3SOX Jun 22, 2021

Choose a reason for hiding this comment

amitojsingh366 Jun 23, 2021

Choose a reason for hiding this comment

amitojsingh366 Jun 23, 2021

Choose a reason for hiding this comment

amitojsingh366 Jun 23, 2021

Choose a reason for hiding this comment

D3SOX commented Jun 22, 2021

D3SOX commented Jun 22, 2021 • edited

BenCarmichael01 commented Jun 22, 2021 • edited

decahedron1 commented Jun 22, 2021 • edited

TimeForANinja commented Jun 22, 2021

BenCarmichael01 commented Jun 22, 2021

D3SOX commented Jun 22, 2021

JeHwanYoo commented Jun 23, 2021 • edited

Zigecek commented Jun 23, 2021

D3SOX commented Jun 23, 2021 • edited

D3SOX commented Jun 23, 2021 • edited

AlbinGrans commented Jun 23, 2021

1001Josias left a comment

Choose a reason for hiding this comment

TimeForANinja commented Jun 23, 2021

skick1234 commented Jun 24, 2021 • edited

mpirescarvalho commented Jun 24, 2021 • edited

TimeForANinja commented Jun 24, 2021

D3SOX commented Jun 24, 2021 • edited

TimeForANinja commented Jun 24, 2021

jbelford commented Jun 24, 2021

mome0320 commented Jun 25, 2021

D3SOX commented Jun 25, 2021

Lazy-C0der left a comment

Choose a reason for hiding this comment

omenocal commented Jun 25, 2021

TimeForANinja commented Jun 25, 2021

TimeForANinja commented Jun 26, 2021

github-actions bot commented Jun 26, 2021

Add parameters to `get_video_info` endpoint to fix 404 errors #938

Add parameters to `get_video_info` endpoint to fix 404 errors #938

D3SOX commented Jun 19, 2021 •

edited

D3SOX commented Jun 22, 2021 •

edited

BenCarmichael01 commented Jun 22, 2021 •

edited

decahedron1 commented Jun 22, 2021 •

edited

JeHwanYoo commented Jun 23, 2021 •

edited

D3SOX commented Jun 23, 2021 •

edited

D3SOX commented Jun 23, 2021 •

edited

skick1234 commented Jun 24, 2021 •

edited

mpirescarvalho commented Jun 24, 2021 •

edited

D3SOX commented Jun 24, 2021 •

edited