Create a Zookeeper backed state repository

[devinsba]

We have built a Zookeeper backed state repository. We store the FeatureState (minus Feature value) as JSON in the a ZNode that is named for the feature. If the feature state is changed in Zookeeper, the state repository will see that change automatically and update its internal cache of values.

We have also provided a few more benchmark tests around this state repository and the others that already existed.

@ryangardner can step in to post benchmark results

[ryangardner]

Here are the various state repository benchmarks - I made an abstract base JMH class that will take a configured state repository and test the performance of pulling out state from the repo - one case for something where the state was not actually set in the state repository to measure that case.

These tests are measuring the steady-state where the state is being actively changed - which is the most common case for something like feature togglz that tend to get changed infrequently.

Benchmark                                                                   Mode  Cnt    Score    Error  Units
PropertyBasedStateRepositoryBenchmark.stateNotSet                          thrpt   50  723.556 ± 32.695  ops/s
PropertyBasedStateRepositoryBenchmark.stateSetExplicitlyInStateRepository  thrpt   50  164.200 ±  5.506  ops/s 


Benchmark                                                              Mode  Cnt      Score      Error  Units
InMemoryStateRepositoryBenchmark.stateNotSet                          thrpt   50  28623.159 ± 1099.964  ops/s
InMemoryStateRepositoryBenchmark.stateSetExplicitlyInStateRepository  thrpt   50  42391.598 ±  894.781  ops/s


Benchmark                                                             Mode  Cnt      Score      Error  Units
CachingStateRepositoryBenchmark.stateNotSet                          thrpt   50  23024.795 ± 1473.696  ops/s
CachingStateRepositoryBenchmark.stateSetExplicitlyInStateRepository  thrpt   50  19076.682 ±  754.279  ops/s

Benchmark                                                                    Mode  Cnt      Score      Error  Units
ZookeeperBasedStateRepositoryBenchmark.stateNotSet                          thrpt   10  28386.520 ± 1917.849  ops/s
ZookeeperBasedStateRepositoryBenchmark.stateSetExplicitlyInStateRepository  thrpt   10  12137.303 ±  493.948  ops/s  

The Zookeeper state repository performance is better than the property based performance. It isn't as good as the performance of the straight in-memory state repository - but it's pretty close.

It should be capable of gracefully handling connection problems with the zookeeper cluster. A change on one of the apps running with this state repository and the same path in the same zookeeper cluster should propagate almost instantly to the other instances using this state repository.

[chkal]

Hey guys! That's VERY impressive! Great work

I must be honest. I'm not familiar with Zookeeper, so reviewing the code is difficult for me. But it looks like you spent quite some time on working on this. And there is good test coverage, which is very important and the performance is awesome.

So I'll merge this into master now.

Thank you so much for this contribution!