Auto Batch Mode?

[w524e]

Remove the link from a text file assigned by --batch-file (or move to a backup file) when a file completed, and automatically resume the list for incomplete download when the last line of the text is done (instead of quitting, process the batch file again--sorry for my poor english), until the list is empty or no more .part file in the current directory. It would be useful for interrupted batch download due to wifi problem or other reasons. Now i'm using sed with some Windows batch code to do this, but it's a bit troublesome. it would be nice if youtube-dl can do this. thanks

[phihag]

Sounds like a great idea.

[FiloSottile]

Meanwhile, if you restart your batch, youtube-dl will simply skip the already downloaded videos at the beginning of the list.

[w524e]

just tried with build 2012.11.27, it will still connect to youtube, download webpage, and extract information...and get file name base on parameters, if .part with then same file name exist, resume; if file name + ext exist then skip...... correct me if my guess are wrong... but don't you thinks it's unnecessary? and if you happen to change your rename rules, it will download everything again!....

[FiloSottile]

Yeah, it's suboptimal, I pointed you at it only as a workaround.
We can implement your feature, but we can't make the skipping method any better (easily, I'd rather prefer avoiding complex heuristics). Or do you have an idea?

[w524e]

the way youtube-dl doing is fine and very "python" :).
i was thinking of annotating the complete link in the batch file before i used the above method. Just put a # in front and youtube-dl will skip the line, just like the todo list--done that and cross it out.

[Plaque-fcc]

… and if you happen to change
your rename rules, it will download everything again!....

And if they publish their videos with a larger available format which
implies files of larger size (or other names originated), there may
come troubles you definetely did not wish to rush into. The only way to
identify the files/links you've already downloaded is making youtube-dl
«remember» what it did times before, which implies having a database
global at least to whole user account.

Do you actually NEED this, @w524e ?

[FiloSottile]

I think he's asking to keep state not globally, but of the current batch.

[Plaque-fcc]

Then it implies that there each batch task will finish succeeding which
no-one guaranties; this means, youtube-dl have to store some info
between its runtimes. This, IMHO, is a quite implementable thing
generally storing (at least, where the playlist file lies) another
file where youtube-dl can store what it got before (video ID,
filename, format ID and optionally download completeness). This way
the tool can read (if told so) and update the info on downloads,
neither harming format/size nor making dupes.

Hit me where I am wrong.

[w524e]

NO....I need youtube-dl "forget" what it did.
No need datebase. this is what i did:
let's say batchlist.txt has:
http//wxw.youtube.com/watch?v=00000000000
http//wxw.youtube.com/watch?v=11111111111
...
http//wxw.youtube.com/watch?v=zzzzzzzzzzz

first, "youtube-dl.exe -x -y -z --batch-file "batchlist.txt"
maybe some "unable to download..." error happen...
then check the result:
If exist *00000000000.mp4 sed.exe -i "/00000000000/D" batchlist.txt
If exist *11111111111.mp4 sed.exe -i "/11111111111/D" batchlist.txt
...

Again,
IF EXIST .*.part or size of batchlist.txt is not zero goto loop
"youtube-dl.exe -x -y -z --batch-file "batchlist.txt"
until, if %%~za==0 GOTO EOF

if DOS can do that, Python can do it better, right? :D

A better way it's like this:
first, youtube-dl read a line from batchlist.txt, say:
"http://www.youtube.com/watch?V=11111111111"
when download is completed 100%, just append prefix (let's say,"# ") to this line so it becomes
"# http://www.youtube.com/watch?V=11111111111"
and then write it back to batchlist.txt by replacing the original line

When youtube-dl reads it, only read the lines that don't start with "# "

Just a different way to handle the "--batch-file" file...

[Plaque-fcc]

I never asked youtube-dl team to add wget flavours; instead, I prefer
using bash shell scripts (analogue to DOS batch files with kind of
VBS/JS control; I am not interested in DOS/NT). This way lets me
separate functional related to web connectivity and my own requisites
in appropriate locations. I guess you may find (using MS WinNT 6.1+)
some Epic PowerShell useful much more than MS-DOS batch files; at least
when I touched Microsoft PowerShell at Windows 7, I found it quite
mimic to POSIX subsystems.

And yes, you say «cmd can do that, so Python too, indeed». Well,
indeed. Ain't Python a scripting language? It is. Ain't DOS
batch/powershell/bash/sh/etc files some sort of scripts too? They all
are. I can imagine the way I use youtube-dl and other stuff for
gathering info about which video I do possess already, load percentage,
codec IDs, so can let youtube-dl know which and how should it download
further, if any. Yes, this may appear harder to script it up than just
having a «guess-it-yourself-you-blackbox» option, but I've never even
read any other sane proposition on how should youtube-dl act to
implement the manner you request.

And yes, you wrote there how your DOS batch file works. And you
called your '--batch-file' file 'not a playlist' while it actually IS
your local custom playlist, heh. Maybe I confused you with this little
terms mixing I did not produce? Then pardon meh, plz. But tell me, if
your batch file handles your trivial workaround, why asking youtube-dl
team implementing this trivial workaround, already handled pretty fine?

[w524e]

That how youtube-dl calls it, (see --help) :D
<< -a, --batch-file FILE file containing URLs to download ('-' for stdin)>>

the way i did can do the job, but it has limitations :( .

Anyway, re-downloading every ~200k Html file (for youtube) might not be the best way to check skipping for those who has a lot (say 1000+ :D) of to "batch". (*edit by @phihag: snip the zen *)

[phihag]

This discussion is getting way out of hand. The feature like a good idea, the data format and proposed algorithms are sufficiently simple to be worthy of being implemented. Feel free to supply a patch request.