Abstract away chef search for discovery

[josephholsten]

Chef Search is used for three purposes in this cookbook:

• discovering configserver instances for mongos: recipes/mongos.rb:29
• discovering replicaset peers for mongod: definitions/mongodb.rb:203
• discovering shard instances for mongos: definitions/mongodb.rb:231

While chef search is great for a good out-of-the-box experience, it's actually quite poorly suited to the task of configuring a database cluster:

• solr updates not synchronous, so an overloaded chef-server may be many seconds behind actual configuration
• node objects in the chef-server may diverge from actual running instances
• chef solo does not natively support search, requiring chef-solo-search

Additionally, our use of search has other problems:

• hard-coded search queries require nodes to match our expected attributes, which may not suit the needs of some users
• assuming the result of discovery is from chef search causes the code to assume it is receiving chef node objects, which makes it cumbersome to replace with another discovery implementation, or ever to stub for testing

Considering all those issues, we should strive to isolate our use of chef-search for discovery. To tackle this, we'll need to:

Improve Library, Definition & Resource APIs

Pull access to node objects out from library functions, and ultimately out of the mongodb_instance definition so wrapper cookbooks can provide alternative discovery methods. These should be possible while remaining backwards compatible.

This should require refactoring MongoDB::configure_replicaset and MongoDB::configure_shards to accept a mere array of strings as node identifiers. To do this, mongodb_instance will need to extract the correct attributes for each node.

With this done, that logic will need to be pulled up into the cookbooks which call mongodb_instance. At this point, it will probably be useful to create a helper library to abstract this discovery. I suggest we model this library after search_for_nodes from the elasticsearch cookbook, and the discovery cookbook from Heavywater.

Improve cookbook API

Define a set of node attributes which allow overriding discovery for most use cases:

• search by role
• search by tag
• search by arbitrary query
• select fqdn attribute from each matched nodes
• select cloud.local_ipv4 attribute from each matched nodes
• select cloud. public_ipv4 attribute from each matched nodes
• select ipaddress attributes from each matched nodes

I suggest we model our use of attributes after search in the elasticsearch cookbook, which uses:

• node['elasticsearch']['discovery']['search_query'] - which defaults to roles:elasticsearch AND chef_environment:#{node.chef_environment} AND elasticsearch_cluster_name:#{node[:elasticsearch][:cluster][:name]}"
• node['elasticsearch']['discovery']['node_attribute'] - which defaults to cloud.local_ipv4 if it exists, or ipaddress

This implies the following :

• node['elasticsearch']['discovery']['search_query'] - which defaults to roles:elasticsearch AND chef_environment:#{node.chef_environment} AND elasticsearch_cluster_name:#{node[:elasticsearch][:cluster][:name]}"
• node['elasticsearch']['discovery']['node_attribute'] - which defaults to cloud.local_ipv4 if it exists, or ipaddress

[jamesonjlee]

I like this idea as it helps us move the cookbook away from chef-search dependency (I thought chef-zero now deals with this a bit?).

anyways, I am for moving discovery out from being so tightly integrated with the cookbook.

[josephholsten]

Looking at serf, it seems we won't be able to easily use serf for discovery until 0.4 is released, it adds key-value tags which we can use to get the <port, arbiterOnly, buildIndexes, hidden, slaveDelay, priority, tags, votes> attributes we need to configure replicaset members.

[josephholsten]

After significant investigation, I've begun working on a project called discoteq.

Because replicaset configuration requires a number of attributes about each hosts, a simple string array won't do the trick. So discoteq provides service data of the form:

{
    "services": {
        "mongodb": [
            {
                "hostname": "mongodb-001.example.net ",
                "port": 27017,
                "arbiterOnly": false
            },
            {
                "hostname": "mongodb-002.example.net ",
                "port": 27017,
                "arbiterOnly": false
            },
            {
                "hostname": "mongodb-003.example.net ",
                "port": 27017,
                "arbiterOnly": true
            }
        ]
    }
}

I'm still locking down the syntax for configuring the lib. At the moment, it supports a JSON config like the following:

{
  "query_map": {
    "myface-fascade": {
      "type": "explicit",
      "records": [
        {
          "hostname": "myface-lb.example.net"
        }
      ]
    },
    "mongodb": {
      "type": "chef_query",
      "query": "(role:mongodb AND tag:awsm)"
      "attrs": {
        "host: "cloud.private_ipv4",
        "port": "mongodb.port"
      }
    },
    "myface-cache": {
      "type": "chef_role"
    }
  }
}

The above allows for an explicit member set, an arbitrary chef search query with arbitrary node attributes, and a very simple role query (query: role:myface-cache, attributes: host:{node['fqdn']}.

I haven't yet locked down a shorthand syntax for storing in this in node attributes yet.

[jamesonjlee]

chef-zero with client provided attributes first yeh?

[wolf31o2]

I would kill to be able to simply specify the information this cookbook needs to setup a replica set. I am quite able to provide a role which gives the information needed by this cookbook. While trying to integrate this with Coopr, I ran into this search issue. Requiring chef-zero or a chef server isn't really the best answer.

[josephholsten]

Moved to sous-chefs/sc-mongodb#77