YouTube channel pagination limit

[valeriangalliat]

Hi,

I'm trying to download all the videos from this channel. youtube-dl finds only 1048 videos during the pagination extraction, but it looks like this channel has more than that.

In fact the YouTube web pagination seems limited to 1048 videos too, but when changing the sort order, we can see that it don't covers all the videos.

Is there any way to fetch all the videos on a channel with more than 1048 videos? Or at least to try both sort orders to increase the limit to 2096?

[valeriangalliat]

I used this to get the newest 1048 videos, so I can at least extract 2096 videos from a channel by running without and with the patch:

sed -i 's/&sort=da//g' youtube_dl/extractor/youtube.py

I don't think there's an easy way to get all the videos when there is more than 2096 without using the API, so I'm closing.

[electroweak]

I am having the same issue - e.g.:
[youtube:channel] UCK08_B5SZwoEUk2hDPMOijQ: Downloading page #34
[youtube:channel] UCK08_B5SZwoEUk2hDPMOijQ: Downloading page #35
[youtube] Channel UCK08_B5SZwoEUk2hDPMOijQ: Found 1050 videos
[download] Downloading playlist: UCK08_B5SZwoEUk2hDPMOijQ
[youtube:channel] playlist UCK08_B5SZwoEUk2hDPMOijQ: Collected 1050 video ids (downloading 1050 of them)

I need a fix for this!!! - What are you doing with sed?

[valeriangalliat]

Hi @electroweak,

I'm patching youtube-dl's YouTube extractor to change the sort order; by default they add &sort=da to the playlist URL so YouTube returns the ~1048 oldest videos. This sed command removes the sort order (the default on YouTube is to return the ~1048 newest videos).

This requires to run youtube-dl from source though. Typically:

git clone https://github.com/rg3/youtube-dl.git
sed -i 's/&sort=da//g' youtube-dl/youtube_dl/extractor/youtube.py
./youtube-dl/youtube_dl/__main__.py "$THE_CHANNEL_URL"

This will retrieve the ~1048 newest videos, so if you ran previously with the other sorting, you'll get ~2096 videos.

It's better, but I currently don't know a way using web scraping to get the remaining videos (if the channel has more than ~2096 videos). An idea is to contact the channel owner, I believe they have the full list of their videos in the YouTube admin area. Another possibility is to use the YouTube API, but I didn't investigate on this yet.