Make clusters+facets more easily searchable

[mrflip]

Quoting howech

Various of the requirement for cluster chef make point toward having a way to glean from the chef node data exactly which nodes belong to a particular cluster (and facet). This is easy enough to do if you iterate through all of the nodes and filter on the cluster_name and cluster_role attributes:

chris@basqueseed:/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/lib/chef/knife$ knife exec -E 'nodes.all {|n| next unless n.cluster_name == "greeneggs"; puts n.node_name}'
greeneggs-delta-0
greeneggs-delta-1
greeneggs-delta-2
greeneggs-delta-3
greeneggs-delta-4
greeneggs-beta
... lots

Iterating through all of the nodes is a little slow, so I would really like to use searching instead. The problem with this is that for searching, chef indexes all of the nested attributes as if they were top level attributes, making it easy to search for values buried deep within the node structure. This feature frustrates using search as a cluster discovery because of some unfortunate choices of other configuration environments:

chris@basqueseed:~/.ssh$ knife exec -E 'nodes.search("cluster_name:greeneggs") {|n| puts n.node_name}' | sort
chimpmark-master-0
chimpmark-slave-0
chimpmark-slave-1
...
goldencap-nikko-0
goldencap-twscraper-0
...
greeneggs-alpha
greeneggs-beta
greeneggs-delta-0
greeneggs-delta-1
greeneggs-delta-2
greeneggs-delta-3
...

What happened? goldencap and chimpmark are configured to talk to the hbase residing on greeneggs by the hbase.cluster_name attribute. Maybe this was a poor choice of attributes to use to indicate which hbase cluster to talk to (it was my choice, actually). But it brings up a general problem with chef search and attribute collisions that we need to be aware of.

Another feature of chef search is that the node attributes are also indexed by the full path. A nodes node.hbase.cluster_name attribute gets indexed as "cluster_name:" and "hbase_cluster_name:value":

chris@basqueseed:~/.ssh$ knife exec -E 'nodes.search("hbase_cluster_name:greeneggs") {|n| puts "#{n.node_name} #{n.cluster_name} #{n.hbase.cluster_name}" }' | sort
chimpmark-master-0 chimpmark greeneggs
chimpmark-slave-0 chimpmark greeneggs
chimpmark-slave-10 chimpmark greeneggs
chimpmark-slave-11 chimpmark greeneggs
chimpmark-slave-12 chimpmark greeneggs
...
greeneggs-delta-3 greeneggs greeneggs
greeneggs-delta-4 greeneggs greeneggs
greeneggs-delta-5 greeneggs greeneggs
greeneggs-delta-6 greeneggs greeneggs
greeneggs-gamma-0 greeneggs greeneggs
greeneggs-gamma-1 greeneggs greeneggs
greeneggs-gamma-2 greeneggs greeneggs

I propose that cluster chef should mark nodes in an unequivocal way so that it can search for them and get the answer it expects. Basically, this means adding a top level "clusterchef" attribute that contains cluster_name, cluster_facet, and cluster_facet_index. The top level "clusterchef" attribute gives us a unique namespace that we can use for the chef search interface. (note that the top level cluster_name attribute will still be there.)

[mrflip]

(As me)

Sounds good.

Would rather use standardized cluster_chef, not clusterchef and ditch the prefix:

node[:cluster_chef][:cluster]
node[:cluster_chef][:facet]
node[:cluster_chef][:index]

[howech]

It turns out that I didn't rely on this way of setting up the attributes in the version_2 code - that is to say, when I am looking for a definitive answer about which cluster a node belongs to, look to the old variable.

[howech]

Not moving forward with this.