[feature] trust model for .chunkdb

[borkd]

Have you read through available documentation and open Github issues?

Yes

Is this a BUG report, FEATURE request, or a QUESTION? Who is the indended audience?

FEATURE request

What and why?

.chunkdb is written and used during chunkserver startup up to 3.0.104-wip. While this seems to be a prudent way to speed up restarts of chunkserver with lots of chunks, current approach is too optimistic and could lead to data safety issues in certain scenarios, see discussion in #146.

Possible improvements:

• The trust placed in this file should rapidly decrease as a function of time since last full chunk scan and number of chunks on said disk. Please allow the administrator to decide what level of paranoia works for them, up to not using .chunkdb at all.

• Assume the data fed from .chunkdb is tainted, and have master hold delay removal of any extra replicas until full chunkserver scan completes.

• Being able to schedule a periodic chunkserver-side (namespace-independent) scrub of chunks across all disks in the cluster would go a long way to gain even more trust from current and future users. See discussions on Tool to get/locate/correct invalid chunk #106.

[acid-maker]

I like that idea. I'll try to find time to implement it in close future.