Only the first 1050 videos are grabbed from a YouTube channel

[ivan]

https://www.youtube.com/user/drighk/videos has about 8,470 videos but youtube-dl https://www.youtube.com/user/drighk/videos only grabs the first 1050:

# PYTHONPATH=. ./bin/youtube-dl https://www.youtube.com/user/drighk/videos
[youtube:user] drighk: Downloading page #1
[download] Downloading playlist: drighk
[youtube:user] drighk: Downloading page #2
[youtube:user] drighk: Downloading page #3
[youtube:user] drighk: Downloading page #4
[youtube:user] drighk: Downloading page #5
[youtube:user] drighk: Downloading page #6
[youtube:user] drighk: Downloading page #7
[youtube:user] drighk: Downloading page #8
[youtube:user] drighk: Downloading page #9
[youtube:user] drighk: Downloading page #10
[youtube:user] drighk: Downloading page #11
[youtube:user] drighk: Downloading page #12
[youtube:user] drighk: Downloading page #13
[youtube:user] drighk: Downloading page #14
[youtube:user] drighk: Downloading page #15
[youtube:user] drighk: Downloading page #16
[youtube:user] drighk: Downloading page #17
[youtube:user] drighk: Downloading page #18
[youtube:user] drighk: Downloading page #19
[youtube:user] drighk: Downloading page #20
[youtube:user] drighk: Downloading page #21
[youtube:user] drighk: Downloading page #22
[youtube:user] drighk: Downloading page #23
[youtube:user] drighk: Downloading page #24
[youtube:user] drighk: Downloading page #25
[youtube:user] drighk: Downloading page #26
[youtube:user] drighk: Downloading page #27
[youtube:user] drighk: Downloading page #28
[youtube:user] drighk: Downloading page #29
[youtube:user] drighk: Downloading page #30
[youtube:user] drighk: Downloading page #31
[youtube:user] drighk: Downloading page #32
[youtube:user] drighk: Downloading page #33
[youtube:user] drighk: Downloading page #34
[youtube:user] drighk: Downloading page #35
[youtube:user] playlist drighk: Downloading 1050 videos
[download] Downloading video 1 of 1050
[youtube] pY2zXx4MZBI: Downloading webpage
[youtube] pY2zXx4MZBI: Extracting video information
[youtube] pY2zXx4MZBI: Downloading DASH manifest

I suspect this regressed when youtube-dl switched away from the v2 RSS feeds.

[ivan]

Possibly related to https://archive.is/xtusH - I think this issue was deleted because the user was deleted.

[ivan]

I noticed something peculiar. Using YouTube Data API (v3) on this channel,

GET https://www.googleapis.com/youtube/v3/channels?part=contentDetails&forUsername=drighk&key={YOUR_API_KEY}

Google returns a playlist ID,
https://www.youtube.com/playlist?list=UUCreCOjTs1N7Lga-4uyKvZg
from which youtube-dl can grab all ~8,500 videos.

Hopefully there's a way to go from user/channel -> playlist without using their key-auth-required API.

[ivan]

It looks like you can get to a "USER's Uploads" playlist via the "Play" button next the "Uploads" section on a user page like https://www.youtube.com/user/drighk

In the HTML, it's on the line that contains Play<

Edit: some users lack a playlist link on that page but do have one on /user/USER/videos?view=57, e.g. https://www.youtube.com/user/greasysideup/videos?view=57

Some users lack a playlist link on either page, though.

[filipe-marques]

I build a project to address this problem is at: https://github.com/filipe-marques/utils
1 - Save the page you want to list the videos with browser in .html file (sometimes you have, at the end of the page, to press the button Load More)
2 - Go to utils project Python3 folder and execute:
python3 YoutubeUrlParser.py "Name_of_Youtube_html_file.html" ( - saved early)
This will result in a file "Name_of_Youtube_html_file.html.txt"
After execute it, will list all youtube video url in a text file.
After, that you can execute the autodown program:
1 - Go to utils project C++/autodown folder, compile the source code
2 - You must execute autodown on the same directory as youtube-dl.
3 - Execute:
(in linux)
./autodown "Name_of_Youtube_html_file.html.txt"
This will download every youtube url in the file "Name_of_Youtube_html_file.html.txt", automatically !

[musicalman]

Hi all,
First I will comment on the Youtube URL parser mentioned recently and then I will present a partial solution to the 1,050 video cap.
In regards to the URL parser, I was able to get it to work, to a point. The parser works perfectly well if you have a list of all the videos you want, however it doesn't address the problem mentioned in this thread, at least not in my experience. The parser was only able to give me URLs for the videos which were showing on the web page. The problem here is that it is difficult to find more than the first 1,050 videos of a playlist or user.
Now, my partial solution. I tried to find the uploads playlist with sea Monkey, which is a Mozilla browser derived from Firefox, and had no success. I did manage to view the uploads playlist by using Google chrome, but it's finicky. I can't tell if it has to do with HTTPS being in the URL or not, as I accidentally forgot the HTTPS part at one point and couldn't get it to work, but in any case, Chrome is the only browser so far I've tested that I can make this work. Since I can't tell what the turning point is, I'll just list the steps I took to make it work.
I copied the URL from two posts above.
https://www.youtube.com/user/greasysideup/videos?view=57
I replaced greasysideup with the username I wanted. Then I used Chrome to go there. This opens a page with the first 30 videos of the user. Close to the first video title, there was a play button as described, however sometimes I couldn't find it. Scrolling around for a while eventually helped make it appear. When I did find it, it took me to the first video from that user, and also showed the uploads playlist. I haven't tried to see if youtube-dl can grab all the videos, but there's no reason why it shouldn't. I was able to view a playlist of almost 3,000 videos so it does work if you can get that pesky play button to show up.
I'm actually not surprised about having to use Chrome, as I've had issues before with Mozilla browsers not working properly with some web pages and having to use Chrome to get it working. I'm not saying Chrome is the determining factor here, though I do wish I knew what's going on. Could it be different software configurations on different PCs which go global and affect all web browsers on the computer? I suspect so, as my desktop and laptop are running similar versions of most of the software I've installed and yet Youtube pages look differently on both machines. On the desktop it's using the HTML5 interface so far as I can tell where as on the laptop it isn't. I don't know how to test for this though, I just heard about this from other friends who have seen similar things.
Obviously the play button is meant to be there as I hear people talking about it in various places, but it just isn't showing up for a decent amount of people. Hopefully, using Chrome will help those like me who were about to tear their hair out. Ideally there would be a simple way to get the uploads playlist ID for a channel.
All of these variables are giving me a headache. I'm no expert in these matters so I won't be able to help with the technical details of things. I can only offer my personal experiences and talk about what I did, not how it works. I'll leave the "how it works" bits to someone who knows what they're doing...

[dstftw]

Will be fixed in the next version.

[ivan]

Thanks, great fix! I didn't notice a UC identifier could be converted to UU.