[Searchable Remote Index] Add safeguards to ensure a cluster cannot be over-subscribed

[andrross]

A searchable snapshot index consumes some resources in the cluster and also requires some amount of local disk to store cached data. Therefore storage is not infinite and there are limits, beyond which the user experience will degrade to unacceptable levels. The goal of this task is to add safeguards to ensure a reasonable user experience is maintained.

We should implement limits to ensure the cluster stays healthy and cannot be unwittingly pushed into an unusable state. In particular, there are two areas to investigate here:

• (Broken out as separate issue) At search time, if there are enough concurrent searches that there is no capacity remaining in the reserved disk cache, then searches should be rejected with an appropriate error message.
• At searchable snapshot index creation time, we should implement limits and reject the requests if the cluster capacity is exceeded. This can likely be implemented as a ratio of cluster-based disk cache size to total data size in the remote repository.

Note that there is future work to implement features that give more of an "infinite" storage experience (at the cost of higher search latencies), but searchable snapshots as it is implemented today keeps some metadata loaded so that data readily searchable and therefore subject to some limits.

[kotwanikunal]

Digging a bit into this issue -

This can likely be implemented as a ratio of cluster-based disk cache size to total data size in the remote repository.

I think this will have to be a function of sum_of_all(restored_index_size) and sum_of_all * (disk_cache_size).

When restoring an index, we will check if sum_of_all * (restored_index_size) + (requested_restored_index_size) is within the bounds to the ratio of sum_of_all * (disk_cache_size) or total_disk_cache_size.

The size of the remote repository can be deceiving since the user might want to restore only a single index, as well as the fact that there can be multiple repositories within a cluster.

[xiaoshi2013]

good