Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix playlist range optimizations for --playlist-items #14740

Closed
kade-robertson opened this issue Nov 13, 2017 · 2 comments
Closed

Fix playlist range optimizations for --playlist-items #14740

kade-robertson opened this issue Nov 13, 2017 · 2 comments

Comments

@kade-robertson
Copy link

@kade-robertson kade-robertson commented Nov 13, 2017

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.11.06. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.11.06

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

Some logic could be implemented to help prevent downloading massive playlists if certain criteria are met. For example, one channel that I archive has uploaded 2000+ videos, so each time I want to download the newest I have to page through 20 pages of the uploads playlist even though I just want the most recent one.

Looking at how the next page is navigated to, it looks like you wouldn't be able to skip any leading pages, but you can certainly avoid downloading extras after.

The easiest solution would be to use the fact that each page has at most 100 videos on it, so you could calculate the very last video index using what is known from the playlist index specifiers (start, end, item count), and only download up to the last page required, i.e.:

  • --playlist-items 1 would only download the first page of the playlist.
  • --playlist-items 99 would also only download the first page of the playlist.
  • --playlist-start 20 --playlist-items 170 would download the first two pages.
  • --playlist-start 20 --playlist-items 181 would download the first three pages.
  • --playlist-start 101 --playlist-end 345 would download the first four pages.
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Nov 13, 2017

This is already implemented for YouTube. Apart from page is 30 videos not 100.

@dstftw dstftw closed this Nov 13, 2017
@kade-robertson
Copy link
Author

@kade-robertson kade-robertson commented Nov 13, 2017

No it isn't.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--playlist-items', u'1', u'-v', u'https://www.youtube.com/user/MM7Games']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.11.06
[debug] Python version 2.7.12 - Linux-4.13.12-xanmod19-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg n3.2.9, ffprobe n3.2.9
[debug] Proxy map: {}
[youtube:user] MM7Games: Downloading channel page
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading webpage
[download] Downloading playlist: Uploads from MM7Games
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #1
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #2
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #3
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #4
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #5
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #6
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #7
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #8
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #9
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #10
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #11
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #12
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #13
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #14
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #15
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #16
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #17
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #18
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #19
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #20
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #21
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #22
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #23
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #24
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #25
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #26
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #27
[youtube:playlist] UUjB_adDAIxOL8GA4Y4OCt8g: Downloading page #28
[youtube:playlist] playlist Uploads from MM7Games: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] INrvJ02TXQM: Downloading webpage
[youtube] INrvJ02TXQM: Downloading video info webpage
[youtube] INrvJ02TXQM: Extracting video information
[youtube] INrvJ02TXQM: Downloading MPD manifest

Clearly there are 28 pages downloaded for a user with 2844 videos uploaded.

dstftw added a commit that referenced this issue Nov 13, 2017
@dstftw dstftw changed the title [YouTube] Avoid fetching entire playlist when downloading partial playlists. Fix playlist range optimizations for --playlist-items Nov 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.