Adaptive recovery

[countvajhula]

While using the jewels scheme (or any storage scheme), we define some parameters at the outset, such as how many shards to use, how many peers to use, and the minimum number of shards for recovery (consequently, the level of redundancy). But in practice, the health of the network may vary widely over time, either due to long term effects like adoption and number of online peers, or due to short term effects like the time of day and internet outages and so on. The optimal parameters for the storage scheme would thus also vary with time.

What the filesystem should provide is not some static guarantee of redundancy factor and number of peers or shards used, but a guarantee relevant to the user experience, such as in terms of availability of their data and the latency in retrieving it. Thus, it may be necessary to dynamically modify the parameters of the storage scheme in order to ensure these guarantees in response to network health, and do so while minimizing the cost (e.g. in terms of redundancy and computational cost).

This feature would have a twofold implementation:

1. Determining the optimal parameters to achieve availability and latency guarantees in terms of measured health parameters (to be defined) of the network (it may be possible to use some form of convex optimization for this).
2. Adapting the existing storage scheme to the new parameters dynamically, while avoiding the need to redo work as much as possible.

Closely related to #7 .