File deduplication mechanism #3030

Closed
rvrignaud opened this Issue Apr 19, 2013 · 4 comments

Comments

Projects
None yet
4 participants
@rvrignaud

Hello,

Does a deduplication files mechanism implementation has been considered ?
What I mean is if I have multiple users that upload the same file it would be great if I could store only once the file (even if it's asynchronous mechanism).
My personal use case is that I run an owncloud instance for a lot of friends and we upload many time same pictures / video. So I have a lot of duplicated files on my server.
It would be very usefull for me to get this feature on application level and not on FS level (as ZFS would be able to give me that ability).

Any thoughts ?

Regards,

@VicDeo

This comment has been minimized.

Show comment
Hide comment
Member

VicDeo commented Apr 19, 2013

related to owncloud/apps#903

@davidak

This comment has been minimized.

Show comment
Hide comment
@davidak

davidak Apr 19, 2013

Member

@rvrignaud why not using existing functionality of a FS?
like wikipedia says btrfs had this too, but don't know it is usable right now.

i think you get a better performance than coding such function in php.

Member

davidak commented Apr 19, 2013

@rvrignaud why not using existing functionality of a FS?
like wikipedia says btrfs had this too, but don't know it is usable right now.

i think you get a better performance than coding such function in php.

@karlitschek

This comment has been minimized.

Show comment
Hide comment
@karlitschek

karlitschek Apr 19, 2013

Member

Yes. I suggest to use the functionality of the OS/Filesystem/Storagebackend. Nothing that should be done in ownCloud.

Member

karlitschek commented Apr 19, 2013

Yes. I suggest to use the functionality of the OS/Filesystem/Storagebackend. Nothing that should be done in ownCloud.

@rvrignaud

This comment has been minimized.

Show comment
Hide comment
@rvrignaud

rvrignaud Apr 19, 2013

@davidak : As far as I know btrfs is not considered as production ready (https://btrfs.wiki.kernel.org/index.php/FAQ#Is_btrfs_stable.3F) and I'm not sure btrfs fully support deduplication (https://btrfs.wiki.kernel.org/index.php/Deduplication).
And AFAIK there is no easy way to use ZFS on linux box.
There are maybe other FS with deduplication feature that I don't know.

@karlitschek : backuppc (http://backuppc.sourceforge.net/) have a deduplication mechanism (http://stanlarson.com/wordpress/?p=118) and I don' t see why this shouldn't/couldn't be done on application layer.

@davidak : As far as I know btrfs is not considered as production ready (https://btrfs.wiki.kernel.org/index.php/FAQ#Is_btrfs_stable.3F) and I'm not sure btrfs fully support deduplication (https://btrfs.wiki.kernel.org/index.php/Deduplication).
And AFAIK there is no easy way to use ZFS on linux box.
There are maybe other FS with deduplication feature that I don't know.

@karlitschek : backuppc (http://backuppc.sourceforge.net/) have a deduplication mechanism (http://stanlarson.com/wordpress/?p=118) and I don' t see why this shouldn't/couldn't be done on application layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment