Filter files that do not match a size predicate

[sarnobat]

With find you can ignore files that are of little consequence, such as files that are really small or really big:

# Files more than 1 gigabyte
find -size +1G

# Files less than 1 megabyte
find -size -1M

This would be great for duff because when trying to free up disk space one wants to find the big files (e.g. videos) without the output being flooded by static web content (e.g. jquery-1.9.2.js, bootstrap.css).

Hopefully that isn't too tough to implement, unlike sorting which would be great but probably algorithmically prohibitive. Being able to use duff with pipes to do things like filtering would be even smarter but I don't see a way to do it the way duff reports (except with the -e option which is a bit risky).

[elmindreda]

It'd be silly for duff to replicate all the options not only in find but all other file-finding tools. That's why you can use duff as a filter. There are even examples of this in the manpage. To use one of your examples above:

find -size +1G | duff

If your file names contain whitespace, you need to use null-termination:

find -size +1G -print0 | duff -0 | xargs -n1 -0 echo

The customizable header (-f) is intended to help with the building of pipes at the other end. For example, if you set it to just output the number of files in each cluster, you get data easily iterable from scripts. See the bundled join-duplicates.sh for an example of this.

[sarnobat]

You're right, this is easily pipable. I'm closing this ticket.

I must have been thinking of some other non-pipable functionality. I do have one question (probably not supposed to be in this ticket) - how does duff buffer the list of files that are fed in from find? Is it by number of bytes, and does the shell control that rather than duff? What I do know is that duff doesn't buffer the entire input before beginning the duplicate detection - thankfully. Though as a consequence can't identify all duplicates.

[elmindreda]

Duff processes files a little and performs an initial sort as they come in, but most of the work cannot be started until all files have been collected, exactly because it does report all duplicates.

[sarnobat]

Oh, I didn't know it batches them all first. I thought I've seen behavior where the buffering prevented the full report and that it streams results instead. I must be wrong.

[elmindreda]

The process_args function provides a nice high-level overview of what duff does and in what order.