why is nothing getting deduped

[mac-linux-free]

Kernel processed data (excludes target files): 49.6G
Comparison of extent info shows a net change in shared extents of: 0.0

with bedup i had 20gb of recovered space.

what could i do ?

[markfasheh]

I'm getting the same thing, on my end it looks like the estimate done for shared extents is going wrong somehow. If you use the show-shared-extents program on a file which was supposed to be deduped you should see some of the extents as shared, which would indicate success. I'll update once I have an idea what's going wrong in duperemove.

[markfasheh]

what kernel are you running? Btrfs on my 3.19 and 3.18 kernels are always reporting extents as shared, which I believe to be a bug.

[mac-linux-free]

yes on my system kernel 3.19.3 is running with btrfs-progs 3.19.1

[markfasheh]

Yeah so I can confirm that fiemap is broken in btrfs from 3.18 and onward - it is reporting shared extents when they are not actually shared. That messes up the estimate duperemove does on change in shared space (we fiemap before and after a dedupe to see how many bytes changed).

You can either roll back to 3.17 or wait until I get a fix out. In the meantime assume the space reporting is broken due to the above issue, but duperemove seems to be doing the right thing.

[mac-linux-free]

thank you very much ... i wait :)

[dominicraf]

This may explain my experience too (issue #51) - although I also found that the estimated free space reported by btrfs fi usage didn't increase after using duperemove. Still, will look forward to the updated duperemove...

[mac-linux-free]

same with kernel 4.0.0

[markfasheh]

I sent a patch out to the btrfs list regarding this today.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/44579

So hopefully this will get sorted out upstream at least. If it is accepted I will ask that it be backported to 3.18 and up.

[mac-linux-free]

kernel 4.0.2 many btrfs patches...but issue not solved

[markfasheh]

Ok, there's two bugs at play here. The first one, regarding btrfs fiemap being broken on kernels 3.18+ is solved by the patch I posted in my previous comment and again tot he btrfs list this week:

http://article.gmane.org/gmane.comp.file-systems.btrfs/45282/match=clear+ret+btrfs_check_shared

We're just waiting on it to be included in the kernel and backported to -stable. From there your distro will have to pick it up. If you build your own kernel things are faster - you can just grab the patch from my e-mail and apply it.

The other bug is detailed in issue#51 and I believe that you suffer from it as well. Fixing it is a work in progress. I'm going to close this bug since I have a patch upstream for it and ask that you follow / comment on issue #51 for the 2nd fix.

Thanks

[mac-linux-free]

does someone know if this is fixed now in kernel 4.1 and btrfs-progs 4.1?

I have different results, for me it is working if i create a new btrfs pool. But after 2 rsync´s --inplace it dedupes again nothing.

[markfasheh]

The fiemap fix should be in 4.1.

The other fixes are on the btrfs mailing list - see issue#51 for a status there. If you feel you have a problem that isn't covered by either of these, let me know.