Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change storage to use LevelDB #12

Merged
merged 53 commits into from
Jan 24, 2019
Merged

Change storage to use LevelDB #12

merged 53 commits into from
Jan 24, 2019

Conversation

hltbra
Copy link
Contributor

@hltbra hltbra commented Jan 23, 2019

I migrated all of the storage to use LevelDB. The results are significantly better and I am ready to try this on a couple of production systems.

I have tried different backends before and they didn't perform better than what we had. I picked LevelDB because it's a small library and a lot simpler than RockDB (fewer features, fewer options, smaller in size). I am definitely considering RocksDB if we think the performance can be better with the usage of column families instead of one keyspace with every value in it.


With this pull request, there's one LevelDB database per Redis database (16 total).
Each data structure has at least one key on LevelDB:

type number of keys in leveldb
string 1
hash 1 + 1 key per field
set 1 + 1 key per element
sorted set 1 + 1 per score + 1 per element
type data it stores
string key value
hash number of elements in the hash
hash field value of the hash field
set number of elements in the set
set element nothing
sorted set number of elements in the sorted set
sorted set element the score of the element
sorted set score nothing

All LevelDB keys have prefixes indicating their type (e.g., hash field, set member, sorted set score). The prefixes were inspired by the blackwidow project 1 2

Examples:

>>> from dredis.ldb import LEVELDB
>>> from dredis.keyspace import Keyspace
>>> LEVELDB.setup_dbs('/tmp/tests')
>>> keyspace = Keyspace()
>>> keyspace.set('mystring', 'stringvalue')
>>> keyspace.hset('myhash', 'field1', 'value1')
1
>>> keyspace.sadd('myset', 'member1')
1
>>> keyspace.zadd('mysortedset', 5, 'element1')
1
>>> keyspace.zadd('mysortedset', 10, 'element2')
1
>>> for db_key, db_value in LEVELDB.get_db('0'):
...     print('%r = %r' % (db_key, db_value))
... 
'\x01\x00\x00\x00\x08mystring' = 'stringvalue'
'\x02\x00\x00\x00\x05myset' = '1'
'\x03\x00\x00\x00\x05mysetmember1' = ''
'\x04\x00\x00\x00\x06myhash' = '1'
'\x05\x00\x00\x00\x06myhashfield1' = 'value1'
'\x06\x00\x00\x00\x0bmysortedset' = '2'
'\x07\x00\x00\x00\x0bmysortedsetelement1' = '5'
'\x07\x00\x00\x00\x0bmysortedsetelement2' = '10'
'\x08\x00\x00\x00\x0bmysortedset@\x14\x00\x00\x00\x00\x00\x00element1' = ''
'\x08\x00\x00\x00\x0bmysortedset@$\x00\x00\x00\x00\x00\x00element2' = ''

@hltbra hltbra mentioned this pull request Jan 23, 2019
@hltbra hltbra merged commit f6f76dd into master Jan 24, 2019
@hltbra hltbra deleted the feature/leveldb branch January 24, 2019 19:47
hltbra added a commit that referenced this pull request Jan 24, 2019
* Change storage to use LevelDB instead of our own directory structure and file formats (#12)
@jimjshields
Copy link

🎆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants