Redis Proto

A multi-threaded implementation of redis written in rust 🦀.

This project is intended to be a drop-in replacement for redis. It’s under construction at the moment.

Design

redis-proto is a black-box multi-threaded re-implementation of redis, backed by tokio. It features data-structure key-space/lock granularity, written entirely in safe rust. It’s currently protocol compatible with redis, so you should be able to test it out with your favourite tools.

The multi-threaded nature has advantages and disadvantages. On one hand, KEYS * isn’t particularly crippling for the server as it’ll just keep a thread busy. On the other hand, there’s some lock-juggling overhead, especially for writes, which messes with tokio.

Building / Running

There’s currently no official release for the project. You can compile and install it yourself with the following command:

cargo install --git https://github.com/obaraelijah/redis-proto

Note: This project requires the rust nightly. You can use rustup to install it.

Once it compiles you should be able to run it with ~ redis-proto.

If you wish to download and run it yourself, you can do the following

~ git clone https://github.com/obaraelijah/redis-proto
~ cd redis-proto
~ cargo run

Then use your favorite redis client. Eg. redis-cli:

~ redis-cli
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> get foo
"bar"

Or using the redis library for python:

import redis
from pprint import pprint

r = redis.Redis()
r.set('foobar', 'foobar')
pprint(r.get('foobar'))

for i in range(100):
    r.rpush('list', i)

list_res = r.lrange('list', 0, -1)

pprint(list_res[0:3])
pprint(sum(map(int, list_res)))

total = 0
for i in range(100):
    total += int(r.lpop('list'))
pprint(total)

Which will print:

b'foobar'
[b'0', b'1', b'2']
4950
4950

Things left to do

Basic Datastructures

• [X] Keys
• [X] Sets
• [X] Lists
• [X] Hashes
• [ ] HyperLogLog
• [ ] Geo
• [-] Sorted Sets
  • [X] Basic Functionality
  • [ ] Still need some operations
• [ ] Strings

Redis Compatibility

• [X] Resp / server
• [ ] Database compatibility
  • [ ] Unsure if this is a good thing – may be better to port existing dumps.
• [ ] Blocking / Concurrent Ops (ttl/save-on-x-ops)
• [ ] CLI / config compatibility
• [ ] Authentication

Contribution Guide

Conduct: =Have fun and be respectful. =

Contact: Make an issue or PR against this repo, or send an email to elijahobara357@gmail.com. If you know of a better forum, please suggest it!

NOTE: DO NOT USE THE REDIS SOURCE CODE IN ANY WAY!

This project is under active development, so things are a little messy.

The general design of redis-proto is:

• A Command (set foo bar) is read off the socket and passed to the translate function in src/ops.rs.
  • The parser generates a RedisValue, which is the lingua franca of redis-proto.
• This gets converted to an Ops::XYZ(XYZOps::Foobar(..)) enum object, which is consumed by the op_interact function.
  • A macro is used to provide automate this.
• This operation is executed against the global State object (using the op_interact function)
  • This will return an ReturnValue type, which is a more convenient form of RedisValue.
  • This ReturnValue is converted and sent back to the client.

Therefore, if you want to do something like implement hashes, you will need to:

1. Add a new struct member in State.
   1. You first define the type: type KeyHash = DashMap<Key, HashMap<Key, Value>>
   2. Then add it to State: pub hashes: KeyHash
2. Define a new file for your data type, src/hashes.rs.
   1. Keep your type definitions in src/types.rs!
3. Create an enum to track your commands, op_variants! { HashOps, HGet(Key, Key), HSet(Key, Key, Value) }
4. Implement parsing for your enum in src/ops.rs.
   1. You should be able to follow the existing parsing infrastructure. Should just be extra entries in translate_array in src/ops.rs.
   2. You will need to add your return type to the ok! macro. Just copy/paste an existing line.
   3. You should return something like ok!(HashOps::HSet(x, y, z)).
   4. A stretch goal is to automate parsing.
5. Implement a async *_interact for your type; I would follow existing implementations (eg. src/keys.rs).
   1. I would keep the redis docs open, and play around with the commands in the web console (or wherever) to determine behavior.
   2. Add a new match entry in the async op_interact function in src/ops.rs.
6. Test it! (follow existing testing bits; eg. src/keys.rs).
7. Please add the commands to the list below.
   1. If you’re using emacs, just fire up the server and evaluate the babel block below (see README.org source)
   2. Alternatively, copy the script into a terminal and copy/paste the output below. (see raw README.org)

Implemented Commands

KeyOps

• Set (Key, Value)
• MSet (RVec<(Key, Value)>)
• Get (Key)
• MGet (RVec<Key>)
• Del (RVec<Key>)
• Rename (Key, Key)
• RenameNx (Key, Key)

ListOps

• LIndex (Key, Index)
• LLen (Key)
• LPop (Key)
• LPush (Key, RVec<Value>)
• LPushX (Key, Value)
• LRange (Key, Index, Index)
• LSet (Key, Index, Value)
• LTrim (Key, Index, Index)
• RPop (Key)
• RPush (Key, RVec<Value>)
• RPushX (Key, Value)
• RPopLPush (Key, Key)
• BLPop (Key, UTimeout)
• BRPop (Key, UTimeout)

HashOps

• HGet (Key, Key)
• HSet (Key, Key, Value)
• HExists (Key, Key)
• HGetAll (Key)
• HMGet (Key, RVec<Key>)
• HKeys (Key)
• HMSet (Key, RVec<(Key, Value)>)
• HIncrBy (Key, Key, Count)
• HLen (Key)
• HDel (Key, RVec<Key>)
• HVals (Key)
• HStrLen (Key, Key)
• HSetNX (Key, Key, Value)

SetOps

• SAdd (Key, RVec<Value>)
• SCard (Key)
• SDiff (RVec<Value>)
• SDiffStore (Key, RVec<Value>)
• SInter (RVec<Value>)
• SInterStore (Key, RVec<Value>)
• SIsMember (Key, Value)
• SMembers (Key)
• SMove (Key, Key, Value)
• SPop (Key, Option<Count>)
• SRandMembers (Key, Option<Count>)
• SRem (Key, RVec<Value>)
• SUnion (RVec<Value>)
• SUnionStore (Key, RVec<Value>)

ZSetOps

• ZAdd (Key, RVec<(Score, Key)>)
• ZRem (Key, RVec<Key>)
• ZRange (Key, Score, Score)
• ZCard (Key)
• ZScore (Key, Key)
• ZPopMax (Key, Count)
• ZPopMin (Key, Count)
• ZRank (Key, Key)

BloomOps

• BInsert (Key, Value)
• BContains (Key, Value)

StackOps

• STPush (Key, Value)
• STPop (Key)
• STPeek (Key)
• STSize (Key)

HyperLogLogOps

• PfAdd (Key, RVec<Value>)
• PfCount (RVec<Key>)
• PfMerge (Key, RVec<Key>)

MiscOps

• Keys ()
• Exists (Vec<Key>)
• Pong ()
• FlushAll ()
• FlushDB ()
• Echo (Value)
• PrintCmds ()
• Select (Index)
• Script (Value)
• EmbeddedScript (Value, Vec<RedisValueRef>)
• Info ()