YT: Batch Archive - Autonumber does not save between sessions.

[ghost]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2019.04.30
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale cp1250, fs mbcs, out cp852, pref cp1250
[debug] youtube-dl version 2019.04.30
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.14393
[debug] exe versions: ffmpeg N-93276-g3b23eb283a, ffprobe N-93276-g3b23eb283a

youtube-dl.exe --format 247+251/247+171/136+140/22/244+251/244+171/135+140/35/243+251/243+171/134+140/43/18 --audio-quality 3 --no-overwrites --abort-on-error --abort-on-unavailable-fragment --merge-output-format mkv --ffmpeg-location "C:\...\ffmpeg-latest-win64-static\bin\ffmpeg.exe" --download-archive log.txt -o "X:\...\%%(autonumber)s_-_%%(title)s_-_%%(uploader_id)s_-_%%(view_count)s_-_%%(id)s_-_%%(vcodec)s_-_%%(acodec)s_-_%%(duration)s_-_%%(upload_date)s.%%(ext)s" "https://www.youtube.com/ ...."

Description

When trying to refresh or restart a partial batch archive, the autonumber incrementation starts from zero, rendering this feature completely useless for such a purpose.

The archive function should be more robust saving a lot more metadata in the log file about the archive, the state of the videos among other things that would help resolve subsequent runs for better consistency (sync).

My situation is that I need the archive function proof to any kind of network issues or interruption and to keep the downloaded archive in perfect order and without any missing files, and doing that by only using youtube-dl would be perfect, without having to rely on any kind of 3rd-party or custom database managing software. I believe with a few more pieces this would work very good without the need of much of anything else, data duplication can easily be done with RAID or PAR or whatever on top or just having two zipped archive copies of the whole thing and testing their hash once in a while.
If I do not use the --abort-on-error or -abort-on-unavailable-fragment, and the downloading of some videos fail because of a temporary local network issue then I would have to restart it and fill those missing files with proper autonumbers manually costing time the software could have done it self easily, so I either have to completely avoid using autonumbers or try many standalone attempts and hope one goes through all the way in one single session, and even that wouldn't work for the subsequent uploads to the channel.

I'd be willing to contribute myself to expand the archive feature, however I'm nowhere near experienced with Python, ... or anytime soon.

There should also be more separation when dealing with the whole channel versus a specific playlist, at least on the outside and terminology so it is more clear when dealing with all the playlists, so that the autonumber is tied fixed to the sorting option used, for example in my case I would want to have it tied to the uploaded date from earliest to latest, first video in the channel gets 0001, the second one gets 0002 and so on and the archive function should force the autonumber to that even if there's a video with the wrong one (redownload or rename), but I guess this is already more suited for another feature request PR which I may do later.

[cfreds]

Try to use only %%(autonumber)s_-%%(title)s-_%% with some videos...

[mockdeep]

I've run into this as well. When downloading a playlist it's pretty common for the connection to drop, so the archive feature is nice in order to zip straight to whichever video it left off on. But then autonumber starts over, so I end up with filenames like:

001 video1.mp4
002 video2.mp4
001 video3.mp4

I thought it would be nice if we could use track_number or episode_number instead, but these all set NA for YouTube playlists. Some sort of playlist_seq would be great. I've tried some of the timestamp options, but release_date and timestamp also give NA, and some playlists are all uploaded on the same date, so upload_date isn't super useful.

[mockdeep]

Nevermind, just found playlist_index via this issue.

[ghost]

Hmm, interesting, I'll look into that, perhaps it's a good workaround until an independent solution comes.