Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement Suggestion: faster dateafter (and datebefore) parameters #5922

Closed
ben569 opened this issue Jun 7, 2015 · 3 comments
Closed

Improvement Suggestion: faster dateafter (and datebefore) parameters #5922

ben569 opened this issue Jun 7, 2015 · 3 comments
Labels

Comments

@ben569
Copy link

@ben569 ben569 commented Jun 7, 2015

I regularly download all the videos from youtube channels. But I only need the most recent videos, I keep the old ones on a server at home. For that, I use the following line in my bash script:
youtube-dl -c --dateafter $date_after --yes-playlist -o '~/Youtube-DL/%(uploader)s/%(title)s.%(ext)s' -f 141/140 --external-downloader aria2c --external-downloader-args '-s 8' $chaine
In $chaine, I usually put URLs along the lines of this: https://youtube.com/user/SOME_YOUTUBER/videos.
Of course, I specify the last time I downloaded the channel in $date_after. By the way, don't hesitate to suggest a cleaner way to eliminate the videos I have already downloaded.
However, since in these URLs the videos are classed by date by youtube, much time is wasted on checking wether or not a video has been posted recently enough, particularly when a channel has several thousands videos.
Maybe this time could be spared by assuming that as soon as a video is found to be out of date, all the remaining are, too.
On a side note, did I setup aria2c right for establishing 8 simultaneous connections? EDIT: I have CN:1 during the downloads.
If needed, I can provide you with my entire script - 23 lines.

@ben569 ben569 changed the title Improvement Suggestion: faster dateafter (and datebefore) options Improvement Suggestion: faster dateafter (and datebefore) parameters Jun 7, 2015
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Jun 8, 2015

If I'm correct, in case of youtube, user pages do not give accurate upload date. (Only inaccurate descriptions like "2 weeks ago"). As a result, it's necessary to extract all videos and compare one by one. Would you like to provide a concrete example URL?

@ben569
Copy link
Author

@ben569 ben569 commented Jun 8, 2015

While the exact upload dates of the videos are not given, the videos on this page are sorted by upload date. As a consequence, youtube-dl could be optimized by checking the upload date of each video (and the other tasks assigned - download, encode, tag...) only until a single video is out of the date range as all the other videos will not make it. It's valid in three cases:

  • Youtube-dl processes the playlist from the lastest to the oldest video and the --dateafter parameter is given. As of the last video within the date range, all the others are too old.
  • Youtube-dl processes the playlist from the oldest to the lastest video and the --datebefore parameter is given. As of the last valid video within the date range, all the others are too recent.
  • Both parameters are given. No matter the way youtube-dl processes the playlist, as of the last video within the date range, all the others are either too recent or too old.

I hope that I was clear enough...

An example URL: https://www.youtube.com/user/MrSuicideSheep/videos. Sorry for the inconvenience, I couldn't think of a youtube channel on top of my head, so I wrote SOME_YOUTUBER so you could swap it with whoever you want.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jan 2, 2016

Duplicate of #1502.

@dstftw dstftw closed this Jan 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.