--download-archive processing order.

[clem16]

Checklist

• I'm reporting a feature request
• I've verified that I'm running youtube-dl version 2019.09.12.1
• I've searched the bugtracker for similar feature requests including closed ones

Description

Ok. I will try to be as concise in describing this as possible. Hopefully it will be understood.

• When I download a playlist it seems to downloads every page for every video in the playlist. - This is quite slow and inefficient in my opinion. If there is a valid programming reason why this is needed, I get it. Not criticizing anyone, just asking / suggesting an alternative way of processing. That could potentially speed up the process, and save bandwidth.

Anyways.

• I have enabled a --download-archive option which writes a list of every video id to a file.

• I would like to see the program check and gather all video ids in a playlist and then match that with a list stored locally and only try to download the missing videos. This should happen before each page is downloaded and checked.

• This would skip the "Downloading pc webpage" message for each and every id in the playlist.

• There would be a huge speedup in processing large playlists.

• This would lighten the load on target servers, some of which I'm sure are already overloaded.

• This would let me add a crontab with more peace of mind, knowing I'm not programmatically hammering the target server on a schedule. I don't run a cron now because of this. I feel doing so is irresponsible and hard on server resources. If it was more efficient I would just be pulling down new videos on a playlist without hammering the server as hard. So this would be good for everyone.

Basically...

• Grab all video ids in playlist, get list of video ids in archive file. Check each id for a match in the archive, if there is no match its a new video and download can begin.

If anyone has any questions or clarifications to what I mean. Just ask and I'll try to clarify my thoughts on what I'd like to see. I'm not familiar enough with the programming / algorithms in this project, or I would fork and try to implement this myself.

Thanks for your time.

[dstftw]

This depends on concrete extractor and may not be technically possible if video id is not available from playlist page.