Restructure node selectors

[davidzhao]

Currently node selectors follow a strict inheritance. For example, RegionAwareSelector is built on top of SystemLoadSelector. This provides limited flexibility if a user wanted to use CPU load or bandwidth as a metric.

I think it would be good to restructure this to be more flexible, where the selection could happen on multiple metrics (even with region as another dimension). For example, you might want to only choose nodes that are using less than X MB/sec bandwidth as well as less than N% CPU.

If anyone's interested in taking this, it'd be great opportunity to design a key piece of the system.

[RamakrishnaChilaka]

I am interested in taking this up.. Kindly, Please check the following PR.. I don't have clarity on the bigger context. Please help and Please give comments by checking the following PR.
#820

[pgokul]

I am looking at picking this up. We can make this more flexible by allowing the node selectors to be chained together. So you could have a simple node selector that looks at bandwidth and another simple node selector that looks at cpu usage. You could build a composite which would combine both. However currently the node selector config is sort of tightly coupled with a single node selector and doesn't lend itself to composition. How should this be handled in the Config ? Should we just add one more key in the yaml say node_selectors and make it an array of node_selector .

[pgokul]

I have submitted a pr - #1535 . The pr has the following changes for restructuring node selectors.
Add support for selecting a node for meeting or for draining( in case we want to scale down a node)
Add support for hard limits for cpu and sysload
Add support for selector chaining. Selectors can now be chained to create more complex selectors out of simple basic ones
Add config changes to allow specifying multiple node selectors to be chained