Playlist response is broken (search & normal playlists)

[K2rlXYZ]

Error:
Playlist response is broken.
In most cases, this error indicates that the problem is on YouTube's side and this is not a bug in the library.
To resolve this error, please wait some time and try again.
If this issue persists, please report it on the project's GitHub page.
YoutubeExplode.Exceptions.TransientFailureException

Code:
static void Main(string[] args) { search(); Thread.Sleep(10000); } private static async void search() { var vids = await new YoutubeClient().Search.GetVideosAsync("random search"); Console.WriteLine(vids.Count); }

[d4n3436]

I have the same problem since yesterday.
Stack trace:

YoutubeExplode.Exceptions.TransientFailureException: Playlist response is broken.
In most cases, this error indicates that the problem is on YouTube's side and this is not a bug in the library.
To resolve this error, please wait some time and try again.
If this issue persists, please report it on the project's GitHub page.
   at YoutubeExplode.ReverseEngineering.Responses.PlaylistResponse.Parse(String raw)
   at YoutubeExplode.ReverseEngineering.Responses.PlaylistResponse.<>c__DisplayClass13_0.<<GetSearchResultsAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at YoutubeExplode.Internal.Retry.WrapAsync[T](Func`1 executeAsync)
   at YoutubeExplode.Internal.Retry.WrapAsync[T](Func`1 executeAsync)
   at YoutubeExplode.ReverseEngineering.Responses.PlaylistResponse.GetSearchResultsAsync(YoutubeHttpClient httpClient, String query, Int32 page)
   at YoutubeExplode.Search.SearchClient.GetVideosAsync(String searchQuery, Int32 startPage, Int32 pageCount)+MoveNext()
   at YoutubeExplode.Search.SearchClient.GetVideosAsync(String searchQuery, Int32 startPage, Int32 pageCount)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at YoutubeExplode.Internal.Extensions.AsyncEnumerableExtensions.ToListAsync[T](IAsyncEnumerable`1 asyncEnumerable)
   at YoutubeExplode.Internal.Extensions.AsyncEnumerableExtensions.ToListAsync[T](IAsyncEnumerable`1 asyncEnumerable)
   at YoutubeExplode.AccessibilityExtensions.BufferAsync(IAsyncEnumerable`1 asyncVideoEnumerable)
   at Fergun.Modules.Utility.YouTube(String query) in /home/d4n/repos/Fergun/src/Modules/Utility.cs:line 1975
   at Discord.Commands.ModuleClassBuilder.<>c__DisplayClass6_0.<<BuildCommand>g__ExecuteCallback|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Discord.Commands.CommandInfo.ExecuteInternalAsync(ICommandContext context, Object[] args, IServiceProvider services)

[YazeedAlKhalaf]

I believe youtube changed something and it has to be reverse-engineered again.
@Tyrrrz

[YazeedAlKhalaf]

https://github.com/Tyrrrz/YoutubeExplode/blob/master/YoutubeExplode/ReverseEngineering/Responses/PlaylistResponse.cs#L145

here is the problem, this URL does not work anymore, I mean the one mentioned in the file above

[t3knical]

it's not broken cause if you know where you put in your own HttpClient on the youtubeClient you can still pull everything just fine, from what I've noticed more importantly is the search function is what's broken (var video = await youtube.Search.GetVideosAsync(song).BufferAsync(maxVideoCount); <-broken), haven't had much time to dig yet but i was able to fix my errors and the demo's errors by using my own custom HttpClient with cookies to both getting the video info and the video stream info's and all seems to work pretty good again, my guess is I'll have to dig into the code to see where it's performing the search function, because my guess is there is an internal youtubeClient I need to setup with cookies to perform raw searches, so if my users aren't searching by link or id and just using simple text it wont work at the moment, it has to be either the link or ID, i'll look more into what the search function does and see if i can fix the whole thing up, but this is 100% a cookies issue. seems like if you're at the 429 rate limit they now enforce cookies now or you can't even pull a video's info anymore before it use to let you pull the vid info no matter what, and the streams would require cookies, now it seems like it's both.

Edit 1: you can see an example of how i'm pushing cookies to a HttpClient in the "#497" issue thread pic I posted there, like i said there it's far from a perfect method but it's got me back up and running just fine.

Edit 2: this was my temp fix on the wpf demo that got the videos info pulling again. WPF Demo Fix Example

Edit 3: this was my temp fix on the console demo that got the videos info pulling again. Console Demo Fix Example

[d4n3436]

it's not broken cause if you know where you put in your own HttpClient on the youtubeClient you can still pull everything just fine

The search function is broken, not the other functions.
Cookies won't solve this problem because the url the lib uses to get the videos in Search.GetVideosAsync() now returns 404.
I ran the tests included in the lib and only SearchSpecs fails.

[t3knical]

it's not broken cause if you know where you put in your own HttpClient on the youtubeClient you can still pull everything just fine

The search function is broken, not the other functions.
Cookies won't solve this problem because the url the lib uses to get the videos in Search.GetVideosAsync() now returns 404.
I've runned the tests included in the lib and only SearchSpecs fail.

which is what i was saying, that i'd have to look into that part to fix the search feature, "from what I've noticed more importantly is the search function is what's broken (var video = await youtube.Search.GetVideosAsync(song).BufferAsync(maxVideoCount); <-broken), haven't had much time to dig yet but i was able to fix my errors and the demo's errors by using my own custom HttpClient with cookies to both getting the video info and the video stream info's and all seems to work pretty good again"

Edit: the title stated "Playlist response is broken", which isn't true since you can still load youtube playlists just fine is all i was saying, and that it was just the search function.

[lat74]

Same here(3 days ago).
Search doesn't work
error message: "Playlist response is broken.
In most cases, this error indicates that the problem is on YouTube's side and this is not a bug in the library.
To resolve this error, please wait some time and try again."

[t3knical]

In the mean time I just made a simple search function using youtube's v3 api to return me the first page of results, and with the 5 videoids (usually only needing the 1st one) it gives me back, I can just run em ID's now through youtubeExplode to get the rest of the info like title likes and more importantly the direct audio link to the file :) so at least i have a janky way of doing normal text searches still. still haven't been able to find an alternative to search_ajax

[YazeedAlKhalaf]

is it mentioned how search_ajax was found?
like in YoutubeExplode how was it found

[t3knical]

took the main dev some trial and error from what i've been reading around, so this is just me guessing but i think he found it playing with the official API. Like monitoring all the POST's and GET's and headers and what not that it was performing while he used it.

Edit: been trying different things on this function in PlaylistResponse.cs but nothing yet has worked at least not to the degree I want it to and w/o using a api key.

[eqenkhjin]

I have same problem. Since 2 days ago. https://youtube.com/list_ajax?style=json&action_get_list=1&list=replace_id_eq&index=1&hl=en this get request return error is : RESPONSE ERROR : Request failed: not found (404)

Header is same as x-youtube-client-version is 20200911 and x-youtube-client-name is 56. It is working for over 5 6 months. Now i think it's changed. Any help.

[d4n3436]

is it mentioned how search_ajax was found?
like in YoutubeExplode how was it found

@YazeedAlKhalaf There's some info in #395.

[Tyrrrz]

Looks like they removed that endpoint altogether.

[gunpal5]

I am also facing the same issue!

[Tyrrrz]

At this point, it looks like we'd have to change the approach to use the search page directly, but that comes with the caveat that a number of fields will be inaccessible. (see #502)

Can someone run a traffic sniffer on YouTube mobile apps (on Android/iOS) and see which requests they are sending when you search for videos? It may be possible that they have access to more info, although probably unlikely.

The bottom line seems that we may have to drop:

• Like/Dislike count
• Upload date
• Search keywords

Additionally, it may be a good idea to brace ourselves and also remove them from videos returned by regular playlists in expectation that similar changes will affect that endpoint as well.

[gunpal5]

@Tyrrrz
I am trying to look into this.

[gunpal5]

@Tyrrrz

Android seems to use protobuf to post the search data. I believe it won't be of any help.

Here's a sample search request from Youtube Android App:

`
POST https://youtubei.googleapis.com/youtubei/v1/search?key=AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w HTTP/1.1
Host: youtubei.googleapis.com
Connection: keep-alive
Content-Length: 1114
X-Goog-Visitor-Id: Cgs5X3RFTlNHZkZScyikg4uBBg%3D%3D
Content-Type: application/x-protobuf
X-Goog-Device-Auth: device_id=AP+lc7+mJUVePxwUd0MIy3Gv2wsVY9Y3Nfz4K6NSwOJ7TwfCuneMo2h0Ss2q7ECOLtT7NgtPL0cAa6JFaqHY8cR0lu6pUmJQG+5qynVjKUdffyU/OUfARaialkVfb2AOKrL37bXcvkFrxg,data=AOspG264xGD3,content=AOspG24iO9jFyk8pJ+eemXBBTCsLF7S1pw
X-GOOG-API-FORMAT-VERSION: 2
User-Agent: com.google.android.youtube/16.04.36(Linux; U; Android 7.1.2; en_US; SM-G965N Build/QP1A.190711.020) gzip
Accept-Encoding: gzip, deflate, br

�b samsungj�SM-G965N �� ��16.04.36 � Android ��7.1.2 ��en-US ��IN � � � � � p@ �UU @ �� �� � d �� � � � � �� � �� �CKWDi4EGEhQxMzc5OTYwMTQ2MDYwODM2NDIwMhilg4uBBiiahq0FKImGrQUo0pytBSiZmq0FKMqcrQUo1JGtBSihkq0FKJKfrQUomY6tBSj1ma0FKLWZrQUo4I2tBSjdl60FKJSXrQUox5etBSjdna0FKOCQrQUol5qtBSjfnK0FKKPLrAUo6v-sBSjilK0FKL2ZrQUop4ytBSiWi60FKPicrQUolJGtBSjYnK0FKPP5rAUo2pWtBQ%3D%3D*|CKWDi4EGEhM2NjUzMTcwMjUyMTcyNTMxNzM4GKWDi4EGKKn6_BIomfz8EijS-vwSKJTk_BIojvn8EijG_fwSKKD6_BIoufX8Eiiy9vwSKNf1_BIotPn8Eiij7fwS �� � @ � � �� �
Asia/Calcutta � samsung��8 x 2�� J �
�
�ms� �CoACgh-a-rDtybqLuDlrKwgHHy7NPpj0qyRHPzv8f-QopVnb8nC8JIPNcymXgd2hmxl0G56B5tnBRKt8T4B4EZXGW3RCRmo2A5qQ3kknFl7xgWw_oNcADw1_arcfcdvpD4PGkNBTjB49kP1rIQy3GhYvp4c0eydDEG0zrIWjZQvq3dH2hKy1cmuO1V8fFinJVf_Oc2hFHmcJXTiH7E7DQgPue_Thhuu06CWJW9ERWtzhvS1Xuq47gzofStR2qiyHTvfetXGLemU8KapOmdrLgcfExtYL1rhqqoilTByDHw4b_mN5NvmVaaLB-HkseLHIKQa0YXSRyKXtsBqrlSi5x41z1xIQDnZ57boFPV84LUwCQGhjOg� machine Z � �����youtube-android-pb" machine2�� � �C(�2���� �C(�2���� (�2���� (�2���� (�2���� �C(�2���� �C(�2�� � (�H �P �X � � � � � � 0j6j0j1 � �� �� � � � E6FC+03 � �
`

[d4n3436]

@gunpal5
That url (https://youtubei.googleapis.com/youtubei/v1/search) and https://www.youtube.com/youtubei/v1/search are from the internal YouTube API. I believe both urls return the same data that I found embedded in the search results page (#502), based on the response I got when using this python library.

[gunpal5]

@d4n3436

It appears Youtube internal APIs accept both protocol buffers and json inputs.

I was not able to decode the protobuf data on fiddler, tried several third party plugins.

[d4n3436]

I found some info about reverse engineering the internal Youtube API and decoding the protobuf data here.

The instructions are for YouTube Music and the code is written Objective-C, but the process for YouTube should be similar.

[Montegro]

Might be related to this one, but since this afternoon I can't seem to fetch any playlists at all.
In my code, it crashes right here;

LockControls(); var client = new YoutubeClient(); var playlist = await client.Playlists.GetVideosAsync(SUBSCRIPTION);

This bit's from the subscriptions function I have, but the same rings true for importing a playlist on its own.

[shaked6540]

Same for me, can't fetch playlists at all now

[Tyrrrz]

Yes, looks like playlists are down too. They seem to have removed the endpoint we were using. Embedded player now retrieves playlist videos using a different endpoint. Example: https://www.youtube.com/youtubei/v1/embedded_player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 (not sure how the key is generated). The data it returns is exactly what is shown on the screen which is less than what Video encapsulates currently.

I found some info about reverse engineering the internal Youtube API and decoding the protobuf data here.

The instructions are for YouTube Music and the code is written Objective-C, but the process for YouTube should be similar.

This is interesting. I poked around, but still can't find what kind of data it returns. If it's less or around the same as the raw HTML approach you found in #502 then it's probably easier to just rely on HTML parsing.

It seems we'd have drop a bunch of data from playlist videos, separate them in its own class (i.e. Video -> PlaylistVideo) and release a new major version because this is a breaking change. Among the fields that are missing, I think only upload date is important, but it's better to not have it at all rather than an approximation (e.g. "6 months ago").

[YazeedAlKhalaf]

Yes, looks like playlists are down too. They seem to have removed the endpoint we were using. Embedded player now retrieves playlist videos using a different endpoint. Example: https://www.youtube.com/youtubei/v1/embedded_player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 (not sure how the key is generated). The data it returns is exactly what is shown on the screen which is less than what Video encapsulates currently.

I found some info about reverse engineering the internal Youtube API and decoding the protobuf data here.
The instructions are for YouTube Music and the code is written Objective-C, but the process for YouTube should be similar.

This is interesting. I poked around, but still can't find what kind of data it returns. If it's less or around the same as the raw HTML approach you found in #502 then it's probably easier to just rely on HTML parsing.

It seems we'd have drop a bunch of data from playlist videos, separate them in its own class (i.e. Video -> PlaylistVideo) and release a new major version because this is a breaking change. Among the fields that are missing, I think only upload date is important, but it's better to not have it at all rather than an approximation (e.g. "6 months ago").

The key is INNERTUBE_API_KEY, below is more explanation
The key is not changing now but you can get it from this URL: https://www.youtube.com/sw.js_data
Simple GET request and no need for any edits. there is also https://www.youtube.com/sw.js which I found out about through trial and error. it made me know that the key is INNERTUBE_API_KEY

[bongoSLAP]

Any update on this or a work around to still get the playlist?
Thanks.

[d4n3436]

@jbr1gh7 You can build the PR (#502) from source and use it, although there are some breaking changes.

[markledwich2]

If it helps at all, I have updated our codebase (which has some parts YouTubeExplode code in it) to work without this endpoint.

Inspired by @YazeedAlKhalaf we ended up using the following method:

• get INNERTUBE_API_KEY from sw.js
• load channel page HTML and grab the video browse endpoint params from ytInitialData
• load paginated videos from youtubei/v1/browse

https://github.com/markledwich2/Recfluence/blob/master/App/YtReader/YtWebsite/YtWeb.cs#L191

It gives us quite a bit less info than before and uses more data 😢. We used to load most videos for all 7K+ channels every day, so we have to get a bit more stingy with this change, but it works. Check out https://transparency.tube/ if you are interested in what we are doing with it.

[Jeager2]

Are you incorporating Mark's update to code into an update from Nuget?

[Montegro]

Just looked into which bits of my code might break and everything, and maybe I'm a bit confused and maybe I'm just handling this a bit differently, but when I have a video downloaded in my program, I have a text file amended with a new line inserted with the video ID :-: Upload Date, and I downloaded a couple new videos and checked that text file and they have their proper dates. So it seems the full date is still available as an endpoint attached to the videos themselves, does the playlist parse result in a different video listing missing this endpoint?

[Tyrrrz]

@Montegro

So it seems the full date is still available as an endpoint attached to the videos themselves, does the playlist parse result in a different video listing missing this endpoint?

Yes. The only currently available replacement features date in a form of "3 days ago".

[Tyrrrz]

Released 6.0-alpha with the fix by @d4n3436.
Stable version will be released later.