[DocDB] Make tserver and master stop writing data when a configurable percentage of disk space is left.

[fritshoogland-yugabyte]

Jira Link: DB-815

Description

Currently, when the tserver (and thus master) runs out of diskspace, it continues to write until the diskspace is exhausted and the tserver gets an operating system out of disk space error.

When the out of disk space error is signalled, the tserver crashes. When the tserver is started again, it will find the database in an inconsistent state, and will try to recover, which will hit the out of disk space issue again. This means that normally, in such a situation, the tserver becomes unavailable.

The current situation means that the tserver cannot be recovered from the out of disk space issue once it hits it, unless free disk space is made available. For that, minimally intervention is needed, which is quite probably manual intervention, and until the intervention happens and disk space is made available the tserver is not available.

It would make sense to have a configurable flag that sets the amount of disk space as a percentage of total disk space, which will stop the tserver from performing client writes.
In that way, the data remains available, and in the case of a crash makes it possible to let the tserver recover the data.