Feature Request: Set the number of skips before aborting.

[bschollnick]

With the skip feature enabled, please add the ability to indicate how many skips should be allowed before terminating the run.

This should also be reset if none skip condition occurs.

Scenario:

The download location is up to date, but someone deleted files. Normally this would abort as soon as it saw the first file (which already existed). With this added, it would scan past the first file, and see that the nth file doesn't exist, as long it was within the range specified.

[Hrxn]

If you don't mind me asking, but I'm not sure if I understand that scenario correctly:

[..] but someone deleted files.

What do yo mean? Deleted locally?

[bschollnick]

Yes, sorry locally...

For example, DeviantArt is a fantastic case, since the files are stored sequently.

It's not unusual for some deviantArt collections to be thousands of files (especially for clubs).
But take a collection that has already been downloaded that consists of 1000 files.

We'll say, A1 .. A1000, exist in the club. But the user last downloaded from a few months ago, A200..A1000.

The user only needs to download A1..A199. So --abort-on-skip, as written will work perfectly. As soon as A200 will be detected on the local drive, it'll abort that collection.

But take a different approach, where someone has modified files, or for some reason, older files may have been manipulated (eg Rule 34, users change tags and filenames on a semi-regular basis). (My Rule 34 downloader does this)

The user maybe up to date, but some files maybe missing, or altered (eg filename has changed).

So let's say the threshold was 10 files, and we have the setup above..
So A1..A5 may exist locally, but A6, A7, A9, A12 may not.

If we stop at A6, then we miss A7, A9, etc al. So I worked around this by creating a user configurable threshold for aborting.

• if the file already exists, increment a counter
• If the counter >= Threshold, then abort
• If a file does not exist, zero the counter, and download it
  (of course, reaching the end of the collection, trumps the counter)

So A6..A7 would clear the counter (downloaded).
A8 increments counter, but is skipped
A9 downloads, and clears.
A10..A11 increments (+2)
A12 clears and downloads itself.
A13..A23 increment the counter, and then Aborts on A23, once it exceeds the threshold of A23.

But --abort-on-skip normally would not be usable here, as I read it, since it would have aborted normally at A1 existing. This preventing the download of A6, A7, A9, A12.

The solution existing now, means even if just a small number of files are missing, you would either have figure out the range, or scan the entirety of collection, which could be thousands of files.

Where a threshold of 10, 25, or even 50, would allow you mostly guarantee you were up to date.

Just a suggestion, of course.

[mikf]

The solution existing now, means even if just a small number of files are missing, you would either have figure out the range, or scan the entirety of collection, which could be thousands of files.

Just in case you didn't notice, I've pushed a commit (6ed629f) that should solve your problem.

Changing --abort-on-skip directly isn't something I want to do (backwards compatibility and all that), but you can now set the skip option to something like abort:10 to stop after 10 consecutive skips.
Use -o skip=abort:10 instead of --abort-on-skip if you want to keep using the command-line.