Make --download-archive work for single videos

[reinierkors]

When a video is already in the archive created by --download-archive youtube-dl will still fetch the video and info pages (for YouTube, at least). Since I am downloading a video, the code already has the IDs beforehand and could safely skip those steps.

I'm not sure if there is any reason it would want to check the page again but initially it seems unnecessary to download two pages and then decide to skip the download if it already has the information needed to skip all these steps beforehand (the video ID).

Thanks again!

[dstftw]

When a video is already in the archive created by --download-archive youtube-dl will still fetch the video and info pages (for YouTube, at least)

No, it won't:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--download-archive', u'yt-dl.txt', u'PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re', u'-f', u'worst', u'--write-pages', u'-v']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.02.09
[debug] Git HEAD: 47246ae
[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-77953-gcc83177, ffprobe N-77953-gcc83177, rtmpdump 2.4
[debug] Proxy map: {}
[youtube:playlist] PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re: Downloading webpage
[youtube:playlist] Saving request to PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re_https_-_www.youtube.com_playlistlist=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re.dump
[download] Downloading playlist: ytdl test PL
[youtube:playlist] playlist ytdl test PL: Downloading 3 videos
[download] Downloading video 1 of 3
[download] Download Youtube Playlist with Youtube-dl has already been recorded in archive
[download] Downloading video 2 of 3
[download] Updating youtube-dl has already been recorded in archive
[download] Downloading video 3 of 3
[download] 37 youtube-dl - faster downloads with aria2 (Programothesis by emailatask.com) has already been recorded in archive
[download] Finished downloading playlist: ytdl test PL

[reinierkors]

Yes it does?

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-wiv', u'--add-metadata', u'--download-archive', u'archive', u'--write-description', u'--write-thumbnail', u'-o', u'channels/%(uploader)s/%(title)s - %(id)s/%(title)s-%(id)s.%(ext)s', u'JDchqzNVOoc']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2016.02.05.1
[debug] Python version 2.7.9 - Linux-3.16.7-ckt20-feral-x86_64-with-debian-8.2
[debug] exe versions: avconv 11.4-6, avprobe 11.4-6, ffmpeg 0.8.17-6
[debug] Proxy map: {}
[youtube] JDchqzNVOoc: Downloading webpage
[youtube] JDchqzNVOoc: Downloading video info webpage
[youtube] JDchqzNVOoc: Extracting video information
[youtube] JDchqzNVOoc: Downloading DASH manifest
[youtube] JDchqzNVOoc: Downloading DASH manifest
[download] ASMR Roleplay - My Favorite Shoes (by request) has already been recorded in archive

[reinierkors]

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-wiv', u'--add-metadata', u'--download-archive', u'archive', u'--write-description', u'--write-thumbnail', u'--print-traffic', u'-o', u'channels/%(uploader)s/%(title)s - %(id)s/%(title)s-%(id)s.%(ext)s', u'JDchqzNVOoc']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2016.02.05.1
[debug] Python version 2.7.9 - Linux-3.16.7-ckt20-feral-x86_64-with-debian-8.2
[debug] exe versions: avconv 11.4-6, avprobe 11.4-6, ffmpeg 0.8.17-6
[debug] Proxy map: {}
[youtube] JDchqzNVOoc: Downloading webpage
send: u'GET /watch?v=JDchqzNVOoc&gl=US&hl=en&has_verified=1&bpctr=9999999999 HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Tue, 09 Feb 2016 23:02:42 GMT
header: Server: gwiseguy/2.0
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
header: X-Frame-Options: SAMEORIGIN
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Strict-Transport-Security: max-age=604800
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: Set-Cookie: VISITOR_INFO1_LIVE=nB2SAGZiCvs; path=/; domain=.youtube.com; expires=Mon, 10-Oct-2016 10:55:42 GMT; httponly
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: YSC=dZnjCelS-qQ; path=/; domain=.youtube.com; httponly
header: Alternate-Protocol: 443:quic,p=1
header: Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
header: Connection: close
header: Transfer-Encoding: chunked
[youtube] JDchqzNVOoc: Downloading video info webpage
send: u'GET /get_video_info?&video_id=JDchqzNVOoc&el=info&ps=default&eurl=&gl=US&hl=en HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: YSC=dZnjCelS-qQ; VISITOR_INFO1_LIVE=nB2SAGZiCvs; PREF=f1=50000000&hl=en; s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Tue, 09 Feb 2016 23:02:49 GMT
header: Server: gwiseguy/2.0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Strict-Transport-Security: max-age=604800
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Content-Encoding: gzip
header: Content-Type: application/x-www-form-urlencoded
header: Cache-Control: no-store
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alternate-Protocol: 443:quic,p=1
header: Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
header: Connection: close
header: Transfer-Encoding: chunked
[youtube] JDchqzNVOoc: Extracting video information
[youtube] JDchqzNVOoc: Downloading DASH manifest
send: u'GET /api/manifest/dash/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm_sd_hd_clear%2Cwebm2_sd_hd_clear/id/o-AFDO4T1JOmpr1vPE7TOAbePgnGKx2ZqJHt8HmPzkir0k/expire/1455080562/ip/2a04%3A1980%3A3100%3A1aac%3A290%3Afaff%3Afe70%3Aa1b0/source/youtube/sparams/as%2Chfr%2Cid%2Cip%2Cipbits%2Citag%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cplayback_host%2Crequiressl%2Csource%2Cexpire/pl/29/hfr/1/nh/IgpwcjA0LmFtczE1KgkxMjcuMC4wLjE/upn/2ffW4-RVfxk/ipbits/0/mm/31/playback_host/r4---sn-5hne6n7z.googlevideo.com/fexp/9407191%2C9414810%2C9416126%2C9417058%2C9420452%2C9422596%2C9423661%2C9423662%2C9425346%2C9425423%2C9425730%2C9425745%2C9425790%2C9425946%2C9427600%2C9427773%2C9427866%2C9428766/requiressl/yes/ms/au/mt/1455058921/mv/m/sver/3/itag/0/signature/A69D056097C0411318E9A8B3CD9BF6A64DD62A0E.6F303D6F99E4D74366E55901D930C56C0B15BC85/key/yt6/mn/sn-5hne6n7z HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: manifest.googlevideo.com\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: video/vnd.mpeg.dash.mpd
header: Date: Tue, 09 Feb 2016 23:02:51 GMT
header: Pragma: no-cache
header: Expires: Fri, 01 Jan 1990 00:00:00 GMT
header: Cache-Control: no-cache, must-revalidate
header: Content-Encoding: gzip
header: Server: HTTP server (unknown)
header: X-XSS-Protection: 1; mode=block
header: X-Frame-Options: SAMEORIGIN
header: Alternate-Protocol: 443:quic,p=1
header: Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
header: Connection: close
header: Transfer-Encoding: chunked
[youtube] JDchqzNVOoc: Downloading DASH manifest
send: u'GET /api/manifest/dash/source/youtube/nh/IgpwcjA0LmFtczE1KgkxMjcuMC4wLjE/pl/29/signature/1D5E5D48FBDA5C112E4CB90F402E86A337833718.431B04E285782B26552531DA7FBBD0B3B4D86FEA/fexp/9407191%2C9414810%2C9416126%2C9417058%2C9420452%2C9422596%2C9423661%2C9423662%2C9425346%2C9425423%2C9425730%2C9425745%2C9425790%2C9425946%2C9427600%2C9427773%2C9427866%2C9428766/sparams/as%2Chfr%2Cid%2Cip%2Cipbits%2Citag%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cplayback_host%2Crequiressl%2Csource%2Cexpire/hfr/1/upn/V-PV0aJhB30/id/o-AMzFquevkudzkj0Zph0X5auqjXDB18kRjVAtf_13vEfe/itag/0/requiressl/yes/key/yt6/ip/2a04%3A1980%3A3100%3A1aac%3A290%3Afaff%3Afe70%3Aa1b0/sver/3/expire/1455080569/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm_sd_hd_clear%2Cwebm2_sd_hd_clear/ms/au/mv/m/mt/1455058921/ipbits/0/mn/sn-5hne6n7z/mm/31/playback_host/r4---sn-5hne6n7z.googlevideo.com HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: manifest.googlevideo.com\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: video/vnd.mpeg.dash.mpd
header: Date: Tue, 09 Feb 2016 23:02:56 GMT
header: Pragma: no-cache
header: Expires: Fri, 01 Jan 1990 00:00:00 GMT
header: Cache-Control: no-cache, must-revalidate
header: Content-Encoding: gzip
header: Server: HTTP server (unknown)
header: X-XSS-Protection: 1; mode=block
header: X-Frame-Options: SAMEORIGIN
header: Alternate-Protocol: 443:quic,p=1
header: Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
header: Connection: close
header: Transfer-Encoding: chunked
[download] ASMR Roleplay - My Favorite Shoes (by request) has already been recorded in archive

[yan12125]

What's the content of archive?

[clb92]

I had this exact issue on version 2015.12.13, but after updating to 2016.02.10 it works fine.

[yan12125]

@xDiglett Could you try with version 2016.02.10?

[dstftw]

It downloads video pages due to non-playlist URL. Generally download archive is expected to be used with playlist URLs.

[reinierkors]

I've switched to a database to keep track of video files instead.

I was using youtube-dl version 2016.02.05.1

[vxbinaca]

@xDiglett Try --no-overwrites