Normalize values across backends

[colebrumley]

I noticed that the backend drivers for etcd and consul (haven't looked at ZK or Bolt) behave differently with regard to key values on puts. Since Consul base64 encodes key values by default and etcd doesn't, etcd key values cannot contain many special characters. This defeats the purpose of an abstraction layer IMO, since as it stands now a user needs to write different code to use libkv with different backends.

I propose that libkv should normalize value inputs so that all backends behave identically. If there's extra processing that needs to be done to accommodate different backends, libkv should do that on behalf of the client. Either escaping unusable characters or b64 encoding the whole value on backends that don't do this automatically are my first thoughts, but I'm sure there are plenty of others.

The flip side of my argument is that normalized values for some backends would no longer be plaintext to non-libkv clients, which could cause trouble for some use cases. I still think this is preferable to the raw values that libkv uses now. Perhaps there could be a boolean 'sanitize' toggle in libkv.NewStore() to maintain backwards compatibility?

Just wanted to drop this here to get opinions. If it makes sense and would be accepted, I'll write it up and do a PR.

[abronan]

Hi @colebrumley, thanks for reporting the issue and testing out the lib. Much appreciated. There are still inconsistencies, as of now we lag behind the least exhaustive client and API which is etcd. It's usable as long as you don't go past the limits of the least exhaustive client/API (key/value space format).

boltDB is a special case and we would like to split KVs between local and global scopes. boltDB would go along a levelDB/RocksDB driver for example.

Luckily etcd APIv3 will fix all those issues (see the etcd APIv3 rfc). Amongst the changes that would allow for a better abstraction are:

• Binary keys and values
• Removing the directory/key distinction (goes along with the first point)

We can fix it now in libkv (I can fix the directory/key distinction by doing _really_dirty_things_™) but the code is non trivial to achieve something that should be there by default.

[colebrumley]

Thanks @abronan, that's seems perfectly cromulent to me. In terms of the general approach going forward, is libkv going to stick with the least exhaustive model, or will there eventually be a minimal feature set required of backend drivers?

Feel free to close this issue if you'd like, thanks for the answers!

[abronan]

I think that once we get those subtleties fixed, the classic trio (consul/etcd/zookeeper) is going to be very similar in terms of features without all the convoluted workarounds to make it work. There is a chance to achieve a good abstraction set for those three.

For the rest and if we add more backends, the possibility to vendor stores independently will allow you to choose which set of backends makes the more sense to support based on their features and call supported. I'll make sure I add a compatibility matrix to make this choice more obvious in the future and when I'm going to tag a first release very soon.

Closing this one, feel free to open another issue if you have any more question! Thanks! :)