Allow for heterogeneous hardware clusters

[alerman]

Currently, when settings are set in the accumulo configs for tserver based properties they apply across the cluster. We are moving toward replacing clusters through rolling upgrades, and at any time may have different machine types running tservers at a given time. Right now that causes us to leave hardware idle because we cannot change settings on only a subset of tservers. Settings which we might want to change depending on the hardware a tserver is running on include, but are not limited to: tserver.cache.*.size, tserver.scan.files.open.max, tserver.compaction.major.concurrent.max etc.

The goal of this issue is to allow tservers running on nodes of a different hardware type to use class based overrides for tserver properties.

• Add ability to add a tserver.class. property where the value is a regex that when matched will put the tserver in this class. EG: tserver.class.beefymachines=bigmachine.*
• Add ability to add .tserver.[property] to override only for this class EG beefymachines.tserver.compaction.major.concurrent.max=10
• Add ability to retrieve class of tserver from config shell command. EG: config --tserver bigMachine123 would print class: beefymachines and then the properties as overridden by that class
• Add ability to retrieve config for a class of machine. EG config --tserver-class beefymachines

[alerman]

I would like to work on this, but wanted to have a chance to discuss the approach before I started working it. Would love feedback and suggestions @keith-turner @ctubbsii

[keith-turner]

I would suggest an approach where the property keys do not have to change. For example, a table property can be set at the site, system, namepace, or table layer using the same property key.

One possible way to do this is to add server config layer between system and namespace config layers. This would lead to the following config layers :

• site
• system
• server (support multiple named server config layer)
• namespace
• table

Not sure what the best way to define named server layers is. One option is instance properties in accumulo site. Or general properties that could be stored at the system level. Or a new config prefix that could only be stored in site or system.

[ivakegg]

@keith-turner Your alternative seem reasonable except that it does not allow table specific properties to be different based on server type (e.g. table.scan.max.memory or table.majc.compaction.strategy). Perhaps that would be too much but its worth the discussion.

[keith-turner]

@keith-turner Your alternative seem reasonable except that it does not allow table specific properties to be different based on server type

What I am thinking of would allow that. For example when the prop table.scan.max.memory is requested on the tserver it would check the table layer, namespace layer, server layer, system layer, site layer, and then default layer. Stopping at the first layer where it is found.

[ctubbsii]

You can already change the tserver settings per machine. You just can't do so in a single file. At present, you must deal with this as a configuration management problem during deployment, and deploy different configs to different tservers. For interoperability, Accumulo only requires the instance.* properties to be identical across servers. Other configs can be different based on the hardware.

You may still need to write a custom balancer or volume chooser to favor more beefy machines, though.

[alerman]

@ctubbsii perhaps this issue then could be as simple as adding a way to determine the settings for a specified tserver to the config shell command.

This brings up another issue in that currently our cluster is likely running in a state where many zk mutable properties have been changed. In order to just use separate config files, we would have to go through and scrub for those and ensure every set property is in the configs appropriately (at least for those that are tserver related). Granted, this is our problem, not accumulo's.

It also seems to me that not having a mechanism for dealing with this in the config shell commands leads to ambiguous behavior, or at least changes what the intuitive definition of "site" is. For instance, when connected to one tserver, would config -f output that value in that tservers configuration under the "site" layer? That leads me, as the user, to believe that is for this entire accumulo instance, but that is not the case.

[ctubbsii]

I agree that some improvements to the shell to make the runtime config of a given tserver query-able would be a welcome improvement. It could be added to the API first, and then utilized by the shell.

[dlmarion]

When this is addressed, the DumpConfig command will need to be updated also