Retrying on failed downloads with archiving & compare post-processor

[Infinitay]

I noticed that some folders ended up having .PART files. I reckon that the download failed at some point. That being said, I realized that because I am using archives and skipping via abort, it wasn't attempting to re-download the failed file. I looked around at how to mitigate this and stumble upon the post-processor compare; however, that "requires downloader.*.part = true and extractor.*.skip = false".

Why isn't compare post-processor available for other extractor.*.skip modes such as abort:N? For example,

    gallery-dl \
    --download-archive archive \
    -o extractor.skip="abort:3" \
    -o extractor.postprocessors="[{ "name": "mtime" }, { "name": "compare" }]"

It would be nice if abort wouldn't increment its count for failed files and only for successfully matched files. Or is there another proper way for retrying on failed downloads?

[mikf]

The "proper" way to retry a failed download is to let gallery-dl download it again, i.e. run it without -o extractor.skip="abort:3". It will detect .part files and continue the download from where it left off. A failed download does not get written into an archive, so using one doesn't matter in your case.

A compare post processor is definitely not what you should be using here. It mainly exists to "update" previously, successfully downloaded files in case the current version of a file differs from the old one.

Why isn't compare post-processor available for other extractor.*.skip modes such as abort:N? For example,

Because its implementation relies on a previously downloaded version of a file and its new .part version existing at the same time so it can compare the two, and that would never be the case with any skip value other than false.

There is a compare.equal option for a compare post processor that can be set to abort:N etc, but compare is still not the right tool here.

[Infinitay]

Okay, I won't use compare, thanks for letting me know.

The "proper" way to retry a failed download is to let gallery-dl download it again, i.e. run it without -o extractor.skip="abort:3". It will detect .part files and continue the download from where it left off. A failed download does not get written into an archive, so using one doesn't matter in your case.

Is there no way to still be able to do this behavior of redownloading part files even with abort? Wouldn't not specifying that option continue to check every since post within a given url?

[mikf]

Well, yes, but you need to somehow let gallery-dl go over all the .part files to complete their download, and the simplest way is to let it go through all posts from a given URL.

You only need to do this once, or until all remaining downloads completed successfully, and afterwards you can continue, like before, to use abort:3.

If you can somehow reconstruct the post URLs from the filenames of the partially downloaded files, then that might be an option as well.

[Infinitay]

Come to think of it I'm getting a bit confused.

A failed download does not get written into an archive, so using one doesn't matter in your case.

Not sure why then it still didn't download it because it shouldn't have been in the archive.

EDIT: I think I know why this happened actually. My archive-format is "{subcategory}_{tweet_id}" but I believe the content that failed and resulted in a .part was from a tweet with multiple attachments. I should modify my current format to include the filename to prevent this from happening again. I suppose that's why it wasn't redownloading either? Though you do say that proper way to re-download is to not include the skip option at all.

[mikf]

My archive-format is "{subcategory}_{tweet_id}"

Oh, that might explain several of your problems. This does not account for the fact that a Tweet can have more than one file. Your Twitter collection is most likely incomplete when you've been using this archive format.

I suppose that's why it wasn't redownloading either?

For a multi-file Tweet, suppose the first file failed to download and gallery-dl went ahead downloaded the second one successfully an writes this ID to the archive. It now looks as though all files of that Tweet got downloaded and gallery-dl will therefore not attempt to re-download the first one.

Though you do say that proper way to re-download is to not include the skip option at all.

Using the default "skip": true would have worked if it weren't for your archive format.

[Infinitay]

Using the default "skip": true would have worked if it weren't for your archive format.

Yea, I will rebuild the archive with the updated format. Going to be incomplete if some posts were deleted on any platform but I suppose there's no other way unless I want to bother migrating everything myself with a script which I don't want to bother with since it's a small archive anyways.