Optimize --download-archive

[tukkek]

When a video is already in the archive created by --download-archive youtube-dl will still fetch the video and info pages (for YouTube, at least). Since I am downloading from a playlist, the code already has the IDs beforehand and could safely skip those steps.

I'm not sure if there is any reason it would want to check the page again but initially it seems unnecessary to download two pages and then decide to skip the download if it already has the information needed to skip all these steps beforehand (the video ID).

Thanks again!

[jaimeMF]

That would easily work for Youtube videos and other site where the url contains the id, but this is not true in all sites (for example TED talks). So it will require some additional work.

[tukkek]

Seems to me this could be implemented as InfoExtractor.should_fetch_pages() (default implementation returns true) to be called near here: https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L340 .

Another option is to create a 'skip information fetching' exception and use it (for now) only on the YouTube extractor.

In both cases you maintain current behavior (always fetch pages) for other websites while YouTube can easily skip playlist items already present in --archive-download.

[phihag]

Fixed - at least for YouTube playlists/user profiles/channels/... - as of youtube-dl v2013.11.22.2 . Type youtube-dl -U to update.

[Hajitorus]

I'm noting that in 2014.01.23.4, this issue is present. (That is, youtube-dl will go out and fetch info for ids in download archive.) I can replicate this by using --download-archive, and running youtube-dl ytuser:therealgiantbomb more than once in a row.

[phihag]

@Hajitorus Sorry, we introduced a bug there. Fixed in youtube-dl 2014.01.29.