File deduplication mechanism

[rvrignaud]

Hello,

Does a deduplication files mechanism implementation has been considered ?
What I mean is if I have multiple users that upload the same file it would be great if I could store only once the file (even if it's asynchronous mechanism).
My personal use case is that I run an owncloud instance for a lot of friends and we upload many time same pictures / video. So I have a lot of duplicated files on my server.
It would be very usefull for me to get this feature on application level and not on FS level (as ZFS would be able to give me that ability).

Any thoughts ?

Regards,

[VicDeo]

related to owncloud-archive/apps#903

[davidak]

@rvrignaud why not using existing functionality of a FS?
like wikipedia says btrfs had this too, but don't know it is usable right now.

i think you get a better performance than coding such function in php.

[karlitschek]

Yes. I suggest to use the functionality of the OS/Filesystem/Storagebackend. Nothing that should be done in ownCloud.

[rvrignaud]

@davidak : As far as I know btrfs is not considered as production ready (https://btrfs.wiki.kernel.org/index.php/FAQ#Is_btrfs_stable.3F) and I'm not sure btrfs fully support deduplication (https://btrfs.wiki.kernel.org/index.php/Deduplication).
And AFAIK there is no easy way to use ZFS on linux box.
There are maybe other FS with deduplication feature that I don't know.

@karlitschek : backuppc (http://backuppc.sourceforge.net/) have a deduplication mechanism (http://stanlarson.com/wordpress/?p=118) and I don' t see why this shouldn't/couldn't be done on application layer.