Shuffle limited to first n number of videos

[DeadManIV]

• I carefully read the contribution guidelines and agree to them.
• I checked if the issue/feature exists in the latest version.

The shuffle in background play of playlists currently doesn't shuffle the entire playlist. It shuffles only the first 50(? I'm not actually sure how many) videos. YouTube itself has the same problem. For reference my playlist is: https://www.youtube.com/playlist?list=FL3hm1ymIqH41ZKDnr1TW5Iw

This website fixes the issue: http://youtube-playlist-randomizer.valami.info/playlist3.php?pl=FL3hm1ymIqH41ZKDnr1TW5Iw

Perhaps NewPipe could use a similar solution for shuffling videos?

P.S. Thanks for all your work, NewPipe is an amazing app!

[theScrabi]

Yes we should take care about this. @karyogamy can you confirm this?

[karyogamy]

Currently, the shuffling is done every time a new partial playlist comes in, so that it doesn't use too much bandwidth to download the entire playlist.

FYI, on Youtube, the partial playlist size is 100 items and is about 25kb. From a good connection, this request takes about 500ms. So we can see this may can add up (time-wise, as the request is sequential) if the user wants to open a long playlist.

Perhaps we can create an toggle to let user choose if they want to download the full playlist when starting to play. This should be an enhancement though, not a bug, as Youtube is doing the same as well.

[theScrabi]

Yes this might be good.
Does Youtube tell how mch elements are in a Playlist before downloading it?

[DeadManIV]

I believe YouTube has changed how their shuffle works, because I'm getting videos from all over my playlist now when I hit shuffle.
I was wondering if this feature was still planned for NewPipe?

[Avelina9X]

This is still very much an issue. Any chance on getting on update on this? It is very annoying to listen to the same 100 songs when they are part of a much much larger playlist.

[sven-hoek]

For anyone interested: As a workaround, you can scroll down to the end of the playlist and start playing one of the last songs on background ("Start here when backgrounded"). After that, you can switch to the background player and activate shuffle. It should then shuffle the whole list.
It would be cool, though, if it were possible to shuffle the whole list without scrolling down each time.

[EitvidasPuras]

This is still an issue. I wonder what makes it so difficult to fix. It's really annoying to have to scroll all the way to the end everytime so that the whole playlist would shuffle :/

[PeterHindes]

This might be worked arround by shuffeling an index ahead of time. It could be much longer than most playlists as it would just be integers. Might be a bit of an inefficent way to implement it, but its probably faster than fetching all the videos.
So we have an array of maybe 10,000 int's and shuffle them randomly and then check if the next number in the array is an index of a song in the playlist fetched from youtube. If not we check the next number untill we find a song.

[PeterHindes]

Looks like, having checked out #1943 this is likeley not a useable solution though, as the problem lies in the loader for the background player, not the shuffle function.

[PeterHindes]

Having looked at a playlist with a large number of videos such as this one. Its clear that youtube has a video count attribute exposed. This might be leveraged to initalize blank spots in the playlist untill they are loaded in.

[XiangRongLin]

Following the idea of @PeterHindes of intializing an array of the playlist size and shuffling that, I found a way to get the video at a specific index in a playlist.

Using this playlist as example:
https://www.youtube.com/playlist?list=PLgfrzIaFR9HfrMfEHHA75y6ieaeyBxuTu

You can get a specific video by using the embedded player and adding index=NUMBER to the url, like:
https://www.youtube.com/embed/?listType=playlist&list=PLgfrzIaFR9HfrMfEHHA75y6ieaeyBxuTu&index=600
Note that the index starts at 0, while the video number in the playlist starts at 1.

The normal video url can be found with following css query, where the href attribut is the contains the video id: link[rel*="canonical"]

If I understand the extractor correcly the YoutubeStreamLinkHandlerFactory would need to be changed here, so that it fetches the webpage and parses the html to extract the id.
But this seems like bad design since all linkhandler only parse the url.

[1dotd4]

I've encountered this bug and I'm following the code to see what is happening. It's the first time here but it seems like the overall problem has been solved backwards.

This is a solution that can be applied to #1795 and #5125 as it change radically the behavior of playlists/queue.

From what I've read so far, to fetch efficiently a playlist, it loads the first 100 and it add the composite items to the queue (see MediaSourceManager and ManagedMediaSourcePlaylist in app/src/main/java/org/schabi/newpipe/player/mediasource/ ). Pressing shuffle it will shuffle just the queue, not the whole playlist. When the index reach the end of the queue it will load more items from the playlist in the queue as they are.

I still didn't find the portion of code that load the list from the playlist, but it seems like it already loads all item to the database, or at least the metadata of the number of items on the playlist. This can be used to fetch all the items when needed to create the queue.

the partial playlist size is 100 items and is about 25kb

Previous solutions seems adding more complexity to a simple list problem. Instead what I am suggesting is to generate a queue list of all items of the playlist. If the user desire to shuffle it, it will be a true shuffle. When needed, fetch the correct items according to the shuffled queue.

Recap: Fetch playlist, copy the playlist to the queue, shuffle it if needed, and then fetch all the needed data to play/view the queued playlist.

[Zetagipp]

So if this issue happens because large playlists are split into multiple parts to save bandwidth, why not have it choose a random section and not just the top 100? That way at the very least you aren't listening to the same section every time you load up a playlist.

[ManOnAJetski]

Hey,

Don't know if anyone has been looking into this issue, but I think I've managed to come up with a somewhat hacky optional workaround: ManOnAJetski@5d8e304

I've never done any Andriod dev before so I'm just posting this here to see if this approach could be used for a more elegant solution by a more experienced NewPipe dev.
Looking at the problem more widely, I think the reason why this issue exists is the "page" system used to load items in a playlist, whereby YouTube provides a json response containing 100 stream items + a link to the next page if there is one, so this issue is pretty baked into the extractor at the moment.

Cheers!

[1dotd4]

@Zetagipp

why not have it choose a random section and not just the top 100?

I think the best solution is shuffling the list before getting the list shown. The latter requires bandwidth to fetch all the required data, but not the first one.

I'm currently unable to work on a patch, though.