weed fix loses (some) indices?

[devinrsmith]

I've got an .dat volume with .idx that I thought might be corrupt. So I backed it up and ran

weed fix ...

It produced an idx file that was smaller than the older one, but the weed volume seemed to come up fine. Unfortunately, it seemed like some things (I haven't confirmed if it's just some items or all) returned 404s, whereas the previous index worked. (Against the same .dat file).

[chrislusf]

Can you please attach the new and old idx files here?

[devinrsmith]

I'll send you a message.

[chrislusf]

Thanks for the data. I think it's related to server not cleanly shutdown. So there are one wrongly formed file. The "weed fix" tool read all file one by one. So all following files are missing.

I remember you did some upgrade and it seems related to that. What version were you upgrading from? And how did you stop the weed volume server? Versions after May 13, 2014 will act on the system interruptions and do a clean shutdown.

I am planning to add a functionality to check the volume files upon starting.

[chrislusf]

Was there any replication for this volume 3?

[devinrsmith]

Yes, I had 3 volume servers total, 1 backup per index.

[chrislusf]

Add some fix for this case. The error handling for replicated writing has a bug, which corrupts the .dat file. The fix will remove the wrong way to handle error.

For existing data, I also added some special handing to avoid possible problem. But it seems still I am not able to recover all the data via "weed fix". For now, you would need to upgrade to the latest version and keep using existing .idx files to avoid losing any data.

[chrislusf]

Please try to use this file to fix the issue. https://github.com/chrislusf/weed-fs/blob/master/unmaintained/fix_dat/fix_dat.go