Abstract storage engines

[jvshahid]

This pr create a storage engine interface and modify shards to use the
interface instead. It also adds a B-Tree implementation and benchmark
tool to compare the performance of the different storage engines.

TODO:

• Make the configuration backward compatible. There were some changes to the config file and we need to be backward compatible with old versions of the config
• Add some metadata to shards so we know what their type is when we're opening existing shards

[jvshahid]

Result of running ./benchmark-storage -points=2000000 -batch=1000 -series=100

################ Benchmarking: leveldb
Writing 2000000 points in batches of 1000 points took 6.237193668s (3.118597 microsecond per point)
Querying 2000000 points took 1.384235448s (0.692118 microseconds per point)
Size: 68M
Took 1.692297994s to delete 1000000 points
Took 1.023558666s to compact
Querying 1000000 points took 564.327603ms (0.564328 microseconds per point)
Size: 31M
Writing 1000000 points in batches of 1000 points took 6.237193668s (3.118597 microsecond per point)
Size: 64M
################ Benchmarking: mdb
Writing 2000000 points in batches of 1000 points took 5.868686799s (2.934343 microsecond per point)
Querying 2000000 points took 719.274242ms (0.359637 microseconds per point)
Size: 229M
Took 1.113599639s to delete 1000000 points
Took 246ns to compact
Querying 1000000 points took 359.482954ms (0.359483 microseconds per point)
Size: 229M
Writing 1000000 points in batches of 1000 points took 5.868686799s (2.934343 microsecond per point)
Size: 230M

[MarkusTeufelberger]

You could try RocksDB too as drop-in replacement for LevelDB (it is facebook's LevelDB fork).
https://github.com/DanielMorsing/rocksdb apparently just replaced leveldb with rocksdb in levigo...

Edit:
Check out https://github.com/tecbot/gorocksdb

[chobie]

+1