RFC: Quotas

Quotas in Voldemort

For each store, assign a soft quota and a hard quota

Periodically, have a background thread scan the directories for each store (note, this requires an environment per store model with BDB) and find utilization statistics

When a store reaches a soft limit, we log a warning messages in the log. We also add the store to a list that is exposed through JMX to notify operations.

When a store reaches a hard limit, we quit allowing puts or updates to the store, until utilization goes back down.

Defining quotas

Quotas are specified per store, per cluster. We divide by the number of nodes in cluster to get the maximum per machine quota.

Finding utilization:

With BDB storage engine, traverse all the files in the stores log directory, looking at File.length on each file.

This task is to run on a fixed interval: run it too frequently and we risk introducing contentions on disk. Run it too infrequently, and we risk a situation where a store goes below the limit but puts are still prohibited as the older quota is kept.

Tracking soft quota violators

Initial way to track soft quota violators is to expose, via an MBean, a list of stores that are presently violating their soft quota. We can also monitor per store disk utilization on each machine, and then trend it together with the existing quota (stacking disk utilization on every machine as to add them up) in ingraphs.

Email notification?

Next evolution of this system might use email notification to contact the violators. We can use another Voldemort store to keep track of who has been sent a quota violation notification and when (to avoid the user being DDoS'd by quota violation emails).