--download-archive option slowed by fetching video links of videos that are already in the archive

[spoulson]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2020.07.28
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

➜  /tmp/test youtube-dl -v --download-archive ~/Downloads/youtube/archive.txt https://www.youtube.com/playlist\?list\=PL4KSefoYcY-Jayr-bIJ0dpCvX9UNWbL7f
[debug] System config: []
[debug] User config: [u'-i', u'--merge-output-format', u'mkv', u'--write-info-json', u'--write-thumbnail', u'--add-metadata']
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--download-archive', u'/Users/spoulson/Downloads/youtube/archive.txt', u'https://www.youtube.com/playlist?list=PL4KSefoYcY-Jayr-bIJ0dpCvX9UNWbL7f']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 2.7.16 (CPython) - Darwin-19.6.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube:playlist] PL4KSefoYcY-Jayr-bIJ0dpCvX9UNWbL7f: Downloading webpage
[download] Downloading playlist: Lawn care
[youtube:playlist] playlist Lawn care: Downloading 2 videos
[download] Downloading video 1 of 2
[download] How To Fix An Ugly Lawn | Lawn Care Tips For Beginners has already been recorded in archive
[download] Downloading video 2 of 2
[download] How To Grow Grass From Seed - Lawn Bare Spot Repair has already been recorded in archive
[download] Finished downloading playlist: Lawn care
➜  /tmp/test ls -l
➜  /tmp/test

Description

When using the --download-archive option, youtube-dl does properly download the video content once, then skip it successive times. However, it still appears to fetch the YouTube video link before skipping the video. This causes a performance bottleneck for downloading content from a channel or playlist with lots of videos.

My expectation is that youtube-dl would download the playlists, then exclude the video links cross referenced in the download archive and then only attempt to download the new video links. This would result in a very quick download of only the newly added videos to the playlists.

My use case is to use youtube-dl to download videos on playlists I created on my own channel. Periodically, I run youtube-dl to capture the videos I added to playlists. Each playlist is a tag-like category. Over the years, I've amassed a large number of playlists and some contain a large number of videos. Running youtube-dl against my channel link works to capture all newly added videos, but it takes several minutes to complete even if there are no new videos.

[dstftw]

It does not download any video links as clearly seen from the log.

[spoulson]

It does not download any video links as clearly seen from the log.

The log says what it wants to say, but there is clearly a per-video delay as it iterates through the playlists. Perhaps I’m wrong it’s not downloading anything at all. The underlying issue clearly still remains.

Please consider reopening.