How to prevent duplicate files ?

[SacDin]

How to prevent duplicate files ?
Does seaweedfs store md5 hash of the file which can be later used to prevent duplicate while uploading ?

[chrislusf]

md5 is not fully implemented.

Files uploaded through s3 has md5.

I will leave this issue open until md5 for filer is implemented.

[SacDin]

Will this be only implemented in filer ?
I mean, will this be available only while accessing files via filer or through direct API as well ?

[chrislusf]

I mean, will this be available only while accessing files via filer or through direct API as well ?

What’s the difference? Everything is through API.

[SacDin]

Filer has its own database for storing metadata, right ?
So this md5 hash will be stored in filer db or with file itself ?

[chrislusf]

md5 is stored with filer db.

any preference?

[SacDin]

No, it is alright. I am currently not using filer but will start using thereafter for md5.

[SacDin]

Great, will try shortly. I have one side question, is it possible to retrieve file metadata without filer ?

[chrislusf]

you can check files in volume servers by http head requests.

[SacDin]

I am confused about how to use it to prevent duplicates. For example, I have incoming upload and I want to prevent same file from storing again. How do I know on which path I have to make HEAD request beforehand ?

Is there any API through which I can lookup / search file by MD5 hash ?

[chrislusf]

No. You would need to maintain a MD5 => file mapping yourself.

[SacDin]

Alright, then I will upload using FUSE and create md5 hash by reading file locally myself. Is there any way to access metadata (md5 hash created by seaweed) while using FUSE based local fs ?

If yes, I will not have to read file in memory and generate md5 myself.

[chrislusf]

files written by FUSE do not have md5 because writes can happen randomly anywhere in the file. It is not efficient to always re-calculate the md5 for any updates.

[CoderYellow]

请教一下，看了半天，意思是去重方案要自己维护文件摘要和文件之间的关系吗，能不能做个api，根据md5，sha之类的信息查询文件是否存在，毕竟类似秒传之类的需求还是很常见的

[9cat]

the best thing is the seaweedfs can handle all the duplication itself, however, the file structure kept in the original place. but the only use the one content