-
Notifications
You must be signed in to change notification settings - Fork 583
Voldemort dev roadmap
These are some of the features that the developers at LinkedIn are working on. Please feel free to add another section and add your future requests.
We are adding support to do store level throttling on the basis of (a) disk space utilization (b) request limita) Batch support - Currently we move one primary partition at a time and checkpoint. If the user is willing to take the risk we should support "multiple" partition movements as well [ Important point to remember - More batch tasks = more redirections and there is a tipping point for redirections ]
b) Smarter parallel execution - Execute tasks in parallel should be smarter and choose tasks to execute so as to avoid two disk sweeps happening on the same node. We need generate a dependency graph of the rebalance tasks and execute them slowly.
c) Fix deletes - Make it run at the end instead of in the middle
d) Better Logging - Currently we propagate the message at the lowest level all the way to the Rebalance tool. Instead we should try to make a better progress bar ( "Number of stores completed - 5 / 10" ) and push that information upwards.
e) Disk sweeps on donor - Currently the stealer node goes into REBALANCING_MASTER state and doesn't allow any disk sweeps ( like slop pusher job, etc ) from not taking place. But what about the poor donor node? ( which is where most of the actual work is being done )
f) Admin tasks during rebalancing - We don't want a user to add / delete stores while rebalancing is going on. Currently the admin task will fail only on the stealer nodes. For the same we need to have some sort of "transition state" on all nodes where-in these admin tasks will be prevented.
a) Better naming format - Why name the files <replica_type> when we can name it <primary_partition>_ ( which is also unique )? This way if we ever add the re-balancing optimization to not copy replica files over this would be a literal "no-op".
b) Sending only one copy of data - This is very well possible now that we have chunks on a per-partition-replica basis.
a) Per request based "required-read" / "required-write" - This should be relatively easy to do since it just means we include two new functions get(K, required-read) and put(K, V, required-write ). Besides that the pipeline routed store can easily set these values accordingly.
b) Callback like support for client - Ability to have a callback for your requests.
a) Config change without downtime - So after working with Voldemort for a while you realize that there are some parameters which you like to change without bringing the server down. In other words a sub-class of VoldemortConfig which are accessible via Admin tool / JMX and fall back to default values if not mentioned. Might be a bad practice, but saves a lot of time.