Change document model for wider persistence engine support

[lusis]

This is a big one. I ran into some false assumptions on my part based on using couchdb+futon as the original persistence engine. Too much stuff was being abstracted away by couchdbkit. I was coding around the readability in Futon.

One of my goals was to support at least the following "NoSQL" engines:

• CouchDB
• MongoDB
• Riak
• Redis (possibly)

As such, my key + dump value strategy may actually not be the best way. The main reason is that mongodb doesn't support keys with dots in the name except in _id. This essentially means that I can't blindly dump the results from a plugin into a value column because Mongodb 1) expects it to either be "encoded" or be valid bson. Essentially, MongoDB is trying to create, I think, subdocuments from my dictionaries.

I have a few options:

• Encode all input.
• Convert any keys I come across that contain dots to underscores and blob dumping as current
• Create them as engine-specific attachments
• Not support MongoDB
• Convert any keys and also make them first-class citizens in the document.

Option 1 breaks one of my original design goals that interfaces would be simple to write in any language. I would have to convey that Vogeler converts all data to, say, base64 before insert. This totally breaks ad-hoc data inspection. Non-starter, IMHO.

Option 2 creates quite a bit of overhead. Since I'm accepting unknown input, I would have to traverse the dictionary created by yaml and json libraries, convert offending keys and insert the data.

Option 3 limits the supportable engines and additionally impacts ad-hoc readability of the data.

Option 4 isn't an option just yet. I'm not ready to give up. However I want to solve this issue so I think the best path at this point is the following:

Option 5 is one of the original tickets someone opened. Worth considering.

I need to study the Riak docs for any gotchas like I ran into with MongoDB. I don't think there will be too many. One other issue I ran into with MongoDB was moving from Couch's DB of documents to Mongo's DB hosting a collection made up of documents. Riak buckets should be much closer the CouchDB model for me but Riak isn't as multi-tenant friendly as the other two so I want to really model how a Riak-backed Vogeler store would look.

At that point, I'll have a better idea of how to model it in the most multi-db friendly way.

Considerations for third-party interfaces:

• Rules should be clearly defined. If I convert dots to some other character like underscore, how does extraction know that it wasn't a REAL underscore?

• Force restrictions on data types that say "No dots in keys". If you don't want to convert dots then create an artificial key and make the resultant data valid values.

  If I do this, I need to write custom parsers for ohai, facter and cobbler which I probably should do anyway. Ohai, facter and cobbler outputs are first-class citizens and user provided input either meets the formatting rule or gets dumped.