Benchmark

Benchmark

Environment

Mac mini Mid 2011

• Processor 2.3 GHz Intel Core i5
• Memory 8 GB 1333 MHz DDR3
• Software OS X 10.9.4 (13E28)
• SSD 128 GB

Configuration

• redis: close save and aof

• ssdb: close binlog manually below:

void BinlogQueue::add_log(char type, char cmd, const leveldb::Slice &key){
    tran_seq ++;
    //Binlog log(tran_seq, type, cmd, key);
    //batch.Put(encode_seq_key(tran_seq), log.repr());
}

• leveldb：

compression       = false
block_size        = 32KB
write_buffer_size = 64MB
cache_size        = 500MB

• boltdb:

nosync = true

Use Redis-Benchmark tool

LedisDB with goleveldb

$ redis-benchmark -p  6380   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q                                     

SET: 28868.36 requests per second
GET: 30156.82 requests per second
INCR: 28042.62 requests per second
LPUSH: 25176.23 requests per second
LPOP: 18288.22 requests per second
LPUSH (needed to benchmark LRANGE): 23052.10 requests per second
LRANGE_100 (first 100 elements): 6175.89 requests per second
LRANGE_300 (first 300 elements): 2640.61 requests per second
LRANGE_500 (first 450 elements): 1817.45 requests per second
LRANGE_600 (first 600 elements): 1331.38 requests per second
MSET (10 keys): 14120.31 requests per second

LedisDB with leveldb

$ redis-benchmark -p  6380   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q                                     

SET: 27886.22 requests per second
GET: 21413.28 requests per second
INCR: 25786.49 requests per second
LPUSH: 21691.97 requests per second
LPOP: 20833.33 requests per second
LPUSH (needed to benchmark LRANGE): 24026.91 requests per second
LRANGE_100 (first 100 elements): 7936.51 requests per second
LRANGE_300 (first 300 elements): 2677.95 requests per second
LRANGE_500 (first 450 elements): 2199.35 requests per second
LRANGE_600 (first 600 elements): 1629.99 requests per second
MSET (10 keys): 20080.32 requests per second

LedisDB with rocksdb

$ redis-benchmark -p  6380   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q  

SET: 27654.87 requests per second
GET: 29550.83 requests per second
INCR: 28457.60 requests per second
LPUSH: 26497.09 requests per second
LPOP: 25087.81 requests per second
LPUSH (needed to benchmark LRANGE): 27027.03 requests per second
LRANGE_100 (first 100 elements): 7053.18 requests per second
LRANGE_300 (first 300 elements): 2935.48 requests per second
LRANGE_500 (first 450 elements): 2303.40 requests per second
LRANGE_600 (first 600 elements): 1686.74 requests per second
MSET (10 keys): 15634.77 requests per second

LedisDB with hyperleveldb

$ redis-benchmark -p  6380   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q  

SET: 29137.53 requests per second
GET: 25278.06 requests per second
INCR: 26567.48 requests per second
LPUSH: 29515.94 requests per second
LPOP: 27639.58 requests per second
LPUSH (needed to benchmark LRANGE): 28835.06 requests per second
LRANGE_100 (first 100 elements): 8227.74 requests per second
LRANGE_300 (first 300 elements): 3317.63 requests per second
LRANGE_500 (first 450 elements): 2070.91 requests per second
LRANGE_600 (first 600 elements): 1620.75 requests per second
MSET (10 keys): 18677.62 requests per second

LedisDB with lmdb

$ redis-benchmark -p  6380   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q    

SET: 28818.44 requests per second
GET: 24497.80 requests per second
INCR: 20938.02 requests per second
LPUSH: 22026.43 requests per second
LPOP: 23419.20 requests per second
LPUSH (needed to benchmark LRANGE): 25575.45 requests per second
LRANGE_100 (first 100 elements): 8325.01 requests per second
LRANGE_300 (first 300 elements): 3686.23 requests per second
LRANGE_500 (first 450 elements): 2617.94 requests per second
LRANGE_600 (first 600 elements): 1889.29 requests per second
MSET (10 keys): 20885.55 requests per second

LedisDB with boltdb

$ redis-benchmark -p  6380   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q    

SET: 2609.60 requests per second
GET: 25290.85 requests per second
INCR: 2549.46 requests per second
LPUSH: 1624.11 requests per second
LPOP: 1239.56 requests per second
LPUSH (needed to benchmark LRANGE): 1257.67 requests per second
LRANGE_100 (first 100 elements): 9889.24 requests per second
LRANGE_300 (first 300 elements): 4054.82 requests per second
LRANGE_500 (first 450 elements): 2812.31 requests per second
LRANGE_600 (first 600 elements): 2106.59 requests per second
MSET (10 keys): 1777.15 requests per second

Redis 2.8

$ redis-benchmark -p  6379   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q  

SET: 28785.26 requests per second
GET: 30211.48 requests per second
INCR: 31826.86 requests per second
LPUSH: 32010.24 requests per second
LPOP: 30978.94 requests per second
LPUSH (needed to benchmark LRANGE): 30506.40 requests per second
LRANGE_100 (first 100 elements): 13793.10 requests per second
LRANGE_300 (first 300 elements): 6505.33 requests per second
LRANGE_500 (first 450 elements): 4856.25 requests per second
LRANGE_600 (first 600 elements): 3964.79 requests per second
MSET (10 keys): 26852.85 requests per second

SSDB latest stable

$ redis-benchmark -p 8888   -n 50000 -t set,incr,get,lpush,lpop,lrange,mset -q       

SET: 29832.94 requests per second
GET: 28137.31 requests per second
INCR: 28851.70 requests per second
LPUSH: 29103.61 requests per second
LPOP: 27901.79 requests per second
LPUSH (needed to benchmark LRANGE): 29638.41 requests per second
LRANGE_100 (first 100 elements): 979.76 requests per second
LRANGE_300 (first 300 elements): 355.19 requests per second
LRANGE_500 (first 450 elements): 231.87 requests per second
LRANGE_600 (first 600 elements): 179.98 requests per second
MSET (10 keys): 21729.68 requests per second

Use Ledis-Benchmark tool

LedisDB with goleveldb

$ ledis-benchmark -n 3000 -port=6380

set: 24143.09 requests per second
incr: 25924.69 requests per second
get: 28356.55 requests per second
rpush: 24129.88 requests per second
lrange: 18876.66 requests per second
lrange: 8523.78 requests per second
lrange: 4821.31 requests per second
lpop: 22452.35 requests per second
hset: 25732.27 requests per second
hget: 25563.63 requests per second
hincrby: 23457.92 requests per second
hdel: 21250.97 requests per second
zadd: 19222.70 requests per second
zincrby: 21545.64 requests per second
zrange: 666.01 requests per second
zrangebyscore: 628.49 requests per second
zrevrange: 63.09 requests per second
zrevrangebyscore: 22.90 requests per second
zrem: 23324.34 requests per second

LedisDB with leveldb

$ ledis-benchmark -n 3000 -port=6380        

set: 25670.67 requests per second
incr: 26972.25 requests per second
get: 20356.61 requests per second
rpush: 26212.65 requests per second
lrange: 16392.43 requests per second
lrange: 8398.24 requests per second
lrange: 5059.10 requests per second
lpop: 24098.06 requests per second
hset: 26435.01 requests per second
hget: 19914.47 requests per second
hincrby: 24545.21 requests per second
hdel: 22800.48 requests per second
zadd: 20124.14 requests per second
zincrby: 20933.10 requests per second
zrange: 3356.76 requests per second
zrangebyscore: 3083.32 requests per second
zrevrange: 121.32 requests per second
zrevrangebyscore: 45.43 requests per second
zrem: 21198.36 requests per second

LedisDB with rocksdb

$ ledis-benchmark -n 3000 -port=6380  

set: 25445.18 requests per second
incr: 27024.74 requests per second
get: 27854.57 requests per second
rpush: 25997.50 requests per second
lrange: 18839.88 requests per second
lrange: 8659.91 requests per second
lrange: 5167.70 requests per second
lpop: 23216.47 requests per second
hset: 26075.90 requests per second
hget: 28183.21 requests per second
hincrby: 24291.93 requests per second
hdel: 21984.06 requests per second
zadd: 20445.58 requests per second
zincrby: 22799.16 requests per second
zrange: 5769.69 requests per second
zrangebyscore: 5652.89 requests per second
zrevrange: 133.57 requests per second
zrevrangebyscore: 113.01 requests per second
zrem: 21606.00 requests per second

LedisDB with hyperleveldb

$ ledis-benchmark -n 3000 -port=6380      

set: 26897.13 requests per second
incr: 26133.26 requests per second
get: 21651.17 requests per second
rpush: 25595.54 requests per second
lrange: 17009.30 requests per second
lrange: 7037.65 requests per second
lrange: 4762.64 requests per second
lpop: 21583.59 requests per second
hset: 25608.47 requests per second
hget: 23319.35 requests per second
hincrby: 23062.92 requests per second
hdel: 21364.14 requests per second
zadd: 19372.19 requests per second
zincrby: 25227.29 requests per second
zrange: 3040.68 requests per second
zrangebyscore: 2966.46 requests per second
zrevrange: 109.29 requests per second
zrevrangebyscore: 42.23 requests per second
zrem: 20411.20 requests per second

LedisDB with lmdb

$ ledis-benchmark -n 3000 -port=6380                                                                           

set: 25949.01 requests per second
incr: 26993.77 requests per second
get: 27270.71 requests per second
rpush: 26403.66 requests per second
lrange: 17132.23 requests per second
lrange: 8690.68 requests per second
lrange: 5070.16 requests per second
lpop: 23844.47 requests per second
hset: 26296.16 requests per second
hget: 26484.43 requests per second
hincrby: 24700.65 requests per second
hdel: 24708.48 requests per second
zadd: 23173.40 requests per second
zincrby: 25020.85 requests per second
zrange: 24993.24 requests per second
zrangebyscore: 24063.70 requests per second
zrevrange: 26075.47 requests per second
zrevrangebyscore: 23769.81 requests per second
zrem: 25144.48 requests per second

LedisDB with boltdb

$ ledis-benchmark -n 3000 -port=6380                                                                           

set: 2627.21 requests per second
incr: 2565.72 requests per second
get: 27659.74 requests per second
rpush: 2067.23 requests per second
lrange: 19280.82 requests per second
lrange: 8702.46 requests per second
lrange: 4980.21 requests per second
lpop: 1621.86 requests per second
hset: 2441.69 requests per second
hget: 27605.53 requests per second
hincrby: 2350.96 requests per second
hdel: 2419.91 requests per second
zadd: 2329.33 requests per second
zincrby: 2367.86 requests per second
zrange: 27318.40 requests per second
zrangebyscore: 25946.14 requests per second
zrevrange: 26707.56 requests per second
zrevrangebyscore: 26017.10 requests per second
zrem: 2393.94 requests per second

Redis 2.8

$ ledis-benchmark -n 3000 -port=6379                                                                  

set: 29373.47 requests per second
incr: 29878.24 requests per second
get: 29056.13 requests per second
rpush: 28837.36 requests per second
lrange: 27694.51 requests per second
lrange: 19949.59 requests per second
lrange: 14082.76 requests per second
lpop: 29694.82 requests per second
hset: 29456.31 requests per second
hget: 31020.73 requests per second
hincrby: 29660.13 requests per second
hdel: 29504.82 requests per second
zadd: 29840.35 requests per second
zincrby: 27501.18 requests per second
zrange: 31620.11 requests per second
zrangebyscore: 23824.60 requests per second
zrevrange: 30413.00 requests per second
zrevrangebyscore: 28600.65 requests per second
zrem: 32168.58 requests per second

SSDB latest stable

$ ledis-benchmark -n 3000 -port=8888  

set: 25641.61 requests per second
incr: 26682.50 requests per second
get: 24433.75 requests per second
rpush: 26884.05 requests per second
lrange: 7007.23 requests per second
lrange: 2009.80 requests per second
lrange: 984.30 requests per second
lpop: 25404.18 requests per second
hset: 26525.97 requests per second
hget: 23762.91 requests per second
hincrby: 26346.00 requests per second
hdel: 25853.36 requests per second
zadd: 27018.46 requests per second
zincrby: 21097.96 requests per second
zrange: 6128.86 requests per second
zrangebyscore: 6189.67 requests per second
zrevrange: 260.11 requests per second
zrevrangebyscore: 239.80 requests per second
zrem: 25401.04 requests per second

Conclusions

1. LedisDB backed by leveldb and rocksdb is little slower than Redis or SSDB, some reasons may cause it:

• Go is fast, but not faster than c/c++.
• LedisDB backed by these storage engines uses cgo to call leveldb, a big cost.

2. Rocksdb is not faster than what we expect, maybe the reason of not appropriate configuration. Even of the developers of RocksDB admitted that they don't fully understand the effect of each configuration change.

3. Generally, among the storage engines supported by LedisDB, lmdb performs better than others, espeically zrange* operations.

4. LedisDB backed by HyperLevelDB performs better than by 'leveldb' and 'rocksdb'.

However, LedisDB is still an alternative NoSQL in production for you.