Floyddotnet / duperemove

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

We were submitting every block at blocksize, but we need to submit the last
blocks of a file at the actual length that they are. Otherwise we were
missing dedupe on those blocks and returning EINVAL.

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Each thread frees the corresponding dups list structure, but in doing so
will remove it from the size list. When this happens it corrupts the list
for push_blocks(). Remove it from the size_list in push_blocks() instead so
we don't have this situation later.

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

The idea of find-dupes is a great one - we want to cut down on the number of
extent references placed on disk by building exents out of our dupe blocks
tree.

The problem is that we've never been able to get this to perform reasonably
well and give good dedupe results at the same time. The design doc in our
wiki has the full details but the most relevant excerpt would be:

We're trying to balance at least 3 very important resources:
 - cpu usage
 - memory usage
 - quality of dedupe

Right now we catch all possible extents (100% dedupe quality) at the expense
of a ton of memory and CPU. Turning down the quality in favor of fewer
expended resources tends to get us in situations where the pattern of dedupe
is seemingly random, or we always miss at least some obvious cases (such as
identical files).

We can continue to experiment until we get something that works well -
there's still many options going forward. In the meantime however, the
number of bug reports I have recieved where find-dupes is a severe
performance problem is too high. We want to ensure a smooth user experience,
especially for those with large dedupe sets so make find-dupes optional.

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Give us dupe, deduped, % totals for filerec stats

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

docs: fix extra dash in man page

Signed-off-by: Ian Kelling <ian@iankelling.org>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>
 # Please enter the commit message for your changes. Lines starting

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

We might be fine using this in all builds but it will require testing first
to be sure we don't take a noticable performance hit.

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Signed-off-by: Mark Fasheh <mfasheh@suse.de>

Add C preprocessor flags

Signed-off-by: Mark Fasheh <mfasheh@suse.de>