New configuration system

A place for thoughts on improving configuration.

Configuration files (e.g. server and client properties files) should be stored centrally and retrieved by clients and servers through a metadata store, similar to what is done with clients retrieving the cluster and store definitions currently.

When a client starts up, it should query the server for its settings via this metadata store. For example, the client would ask the server for its timeout property via store→get(“client.timeout”).

Properties can also be arranged (by convention) into namespaces or packages. Clients could take advantage of this feature and retrieve all of their associated properties via the getAll method [ store→getAll(“client”) would retrieve all of the properties in the client namespace].

Unlike the existing metadata store interfaces, the configuration metadata stores should allow retrieval of individual values rather than the entire configuration file. By enabling this mode of operation, individual values could be updated and specified without affecting the remainder of the blob.

To allow for different configurations for different clients, a hierarchy can be introduced to the metadata store. For example, a client could retrieve its metadata from the store (“client/production/host1”). When values are returned to the client, the server will look for the value in the three stores (client/production/host1, client/production, client) in order until the value is found. When values are saved (put) back to the server, the specific metadata store (client/production/host1) is updated.

The Storage Engines for these metadata stores could be configurable if desired. For example, it would be possible to store this metadata in SQL or BDB or flat files. Metadata stores should use the “ALL” routing with “ALL” replicas of the data. The settings for preferred reads and writes should be configurable, based on customer need. Metadata stores should not accept or use custom resolvers but should resolve values based on timestamps.

The same concepts used to retrieve client configurations can also apply to the servers. Servers can also retrieve their metadata (server.properties as well as store and cluster information) via a metadata store. Servers will need some special code for bootstrapping purposes. For bootstrapping purposes, servers should maintain the last known settings for their properties in a flat file (either XML or properties). This file is used by the server to bootstrap its metadata configuration until the current values can be obtained from the cluster. Clients may wish to (but not be required to) use a similar approach for bootstrapping their settings. The client bootstrapping would be used to control timeout and other values until the client could determine the correct values from the servers.