Ability to Disable Downloading Images Older Than Last Rip

[Davrial]

I have downloaded a lot of albums at this point, and I frequently use the "Re-Rip Checked" option to download new content in said albums. The problem is, if there are any images in the albums that I delete (mostly filler images, thumbnail versions, etc), they all get re-downloaded when I re-rip.

So my suggestion (slash question as to if it's even possible) is as what the title says: Would it be possible to add a check/toggle option to disable the downloading of files older than the last rip, specifically when re-ripping albums? That may or may not even make the re-ripping even faster, as it would force it to skip over images older than set date/time, which would mean it wouldnt have to check if a file has already been downloaded.

[rautamiekka]

Unless someone has a better idea, that'd mandate using a file containing the links to the previously downloaded files. For example: https://github.com/rg3/youtube-dl/ can use a file which lists the vid IDs of already fetched vids -after- you DL, which is way easier to implement and use compared to a general pic ripper.

The JSON format I can imagine right now that'll enable matching multiple items from a single address:

{
    "ORIGINAL LINK SUCH AS FOLDER OR ALBUM": {
        "ITEM IN FOLDER OR ALBUM": [
            "DOWNLOAD LINK USED",
            ...
        ], ...
    }, ...
}

For example:

{
    "http://rautamiekka.deviantart.com/gallery/?catpath=/": {
        "http://rautamiekka.deviantart.com/art/Princess-Luna-BIG-COLLAB-colored-0-672156533": [
            "http://img09.deviantart.net/ed89/i/2016/335/4/2/_sleeping_on_the_job___finnish_by_rautamiekka-daq4v9l.png"
        ]
    }, ...
}

That allows for very fine (by folder/album address, item address in the folder/album, the DL link used, the filename used) matching of already-downloaded content, assuming the same format works for everything.

Otherwise we'll need ripper-specific formats built into the specific ripper and an unified API to use the format.

[ghost]

Having this addition to the app would drastically reduce download times.

[rautamiekka]

^ Re-download times, mind you.

[Davrial]

^ *Re-RIP times, mind you. It would be explicitly preventing re-downloads.

Also as for your overspecialization of folder/file details, it wouldnt need to be nearly that complicated. You would just have the ripper document the last time it ran a rip and then have that compare to the metadata on a site which shows its age/timestamp. It would be extra easy with sites like Tumblr where all images are stored with a string name, because then the ripper could just be told to only rip images with a newer string than the most recent image currently saved/ripped for each specific url/folder.

[metaprime]

@rautamiekka That's exactly the solution I imagined. This request has been opened and discussed multiple times in the past, and at least one of those issues is open (probably aggregating links to the duplicates as well).

[ghost]

I support this enhancement as well. I opened an issue for deduplication a week or two ago, but it got closed since "ripme doesn't have deduplication by design".

[metaprime]

@ANonCoder123 This is a distinct issue from deduplication. This is an issue of not downloading URLs already downloaded. It mostly would solve the problem of deduplication (except for duplicates downloaded from different URLs)

[cyian-1756]

This was added, to use it click the "Remember URL history" button in the config menu