Redis sharding proxy
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
FunctionalTest.cc
LICENSE
Main.cc
Makefile
NutcrackerConsistentHashRing.cc
NutcrackerConsistentHashRing.hh
Protocol.cc
Protocol.hh
ProtocolTest.cc
Proxy.cc
Proxy.hh
README
redis-shatter.conf.json
redis.conf
run_multiple_redis.sh
run_tests.sh

README

redis-shatter
-------------

redis-shatter is a sharding proxy for the Redis protocol, inspired by twemproxy
(https://github.com/twitter/twemproxy). Documentation on how to actually use
this is in the example configuration file, redis-shatter.conf.

In essence, this proxy appears like a standard Redis server to other hosts, but
doesn't store any data locally - instead, keys are distributed between multiple
backends (other hosts that speak the Redis protocol, usually Redis servers).
This can give useful speed, reliability, and scalability benefits, but has some
significant drawbacks as well:
- Command execution requires another network hop, which comes with a performance
  penalty. Clients that pipeline commands are somewhat insulated from this
  issue.
- Complex commands that affect multiple keys cannot be run efficiently unless the
  affected keys are all on the same backend. There are inefficient ways to run
  these commands, but these are (currently) not implemented by redis-shatter.


Performance
-----------

redis-shatter in front of 8 Redis server instances on the same machine was about
60% as fast as a single Redis server itself, according to side-by-side
redis-benchmark tests. Very little performance optimization work has been done,
so there's probably a lot of room for improvement here.

Like most of my projects, this is only tested at a small scale (so far), so
there may be unfound bugs or inefficiencies. Use at your own risk.


Behavior
--------

Unlike other similar projects, redis-shatter strives to implement as many
commands as possible. For some of the less-common commands, redis-shatter
deviates from the standard Redis protocol in order to implement reasonable
behavior. These deviations are explained in the notes in the below table.

Command             -- Supported  -- Notes (see end)
--------------------------------------------------
APPEND              -- Yes        --
AUTH                -- No         --
BGREWRITEAOF        -- Yes        -- *6
BGSAVE              -- Yes        -- *6
BITCOUNT            -- Yes        --
BITFIELD            -- Yes        --
BITOP               -- Yes        -- *2
BITPOS              -- Yes        --
BLPOP               -- No         --
BRPOP               -- No         --
BRPOPLPUSH          -- No         --
BZPOPMAX            -- No         --
BZPOPMIN            -- No         --
CLIENT GETNAME      -- Yes        -- *F
CLIENT KILL         -- No         --
CLIENT LIST         -- Yes        -- *C
CLIENT PAUSE        -- No         --
CLIENT REPLY        -- No         --
CLIENT SETNAME      -- Yes        -- *F
CLUSTER             -- No         -- *H
COMMAND             -- Yes        -- *E
COMMAND COUNT       -- Yes        -- *E
COMMAND GETKEYS     -- Yes        -- *E
COMMAND INFO        -- Yes        -- *E
CONFIG GET          -- Yes        -- *5
CONFIG RESETSTAT    -- Yes        -- *6
CONFIG REWRITE      -- Yes        -- *6
CONFIG SET          -- Yes        -- *6
DBSIZE              -- Yes        -- *A
DEBUG OBJECT        -- Yes        --
DEBUG SEGFAULT      -- No         --
DECR                -- Yes        --
DECRBY              -- Yes        --
DEL                 -- Yes        -- *4
DISCARD             -- No         --
DUMP                -- Yes        --
ECHO                -- Yes        --
EVAL                -- Yes        -- *0 *2
EVALSHA             -- Yes        -- *0 *2 *G
EXEC                -- No         --
EXISTS              -- Yes        -- *4
EXPIRE              -- Yes        --
EXPIREAT            -- Yes        --
FLUSHALL            -- Yes        -- *6
FLUSHDB             -- Yes        -- *6 *7
GEOADD              -- Yes        --
GEODIST             -- Yes        --
GEOHASH             -- Yes        --
GEOPOS              -- Yes        --
GEORADIUS           -- Yes        -- *2
GEORADIUSBYMEMBER   -- Yes        -- *2
GET                 -- Yes        --
GETBIT              -- Yes        --
GETRANGE            -- Yes        --
GETSET              -- Yes        --
HDEL                -- Yes        --
HEXISTS             -- Yes        --
HGET                -- Yes        --
HGETALL             -- Yes        --
HINCRBY             -- Yes        --
HINCRBYFLOAT        -- Yes        --
HKEYS               -- Yes        --
HLEN                -- Yes        --
HMGET               -- Yes        --
HMSET               -- Yes        --
HSCAN               -- Yes        --
HSET                -- Yes        --
HSETNX              -- Yes        --
HSTRLEN             -- Yes        --
HVALS               -- Yes        --
INCR                -- Yes        --
INCRBY              -- Yes        --
INCRBYFLOAT         -- Yes        --
INFO                -- Yes        -- *8
KEYS                -- Yes        -- *9 *A
LASTSAVE            -- Yes        -- *5
LINDEX              -- Yes        --
LINSERT             -- Yes        --
LLEN                -- Yes        --
LPOP                -- Yes        --
LPUSH               -- Yes        --
LPUSHX              -- Yes        --
LRANGE              -- Yes        --
LREM                -- Yes        --
LSET                -- Yes        --
LTRIM               -- Yes        --
MEMORY DOCTOR       -- Yes        -- *5
MEMORY HELP         -- Yes        -- *E
MEMORY MALLOC-STATS -- Yes        -- *5
MEMORY PURGE        -- Yes        -- *5
MEMORY STATS        -- Yes        -- *5
MEMORY USAGE        -- Yes        --
MGET                -- Yes        -- *4
MIGRATE             -- Yes        -- *4 *J
MONITOR             -- No         --
MOVE                -- No         --
MSET                -- Yes        -- *4
MSETNX              -- Yes        -- *2
MULTI               -- No         --
OBJECT ENCODING     -- Yes        --
OBJECT FREQ         -- Yes        --
OBJECT IDLETIME     -- Yes        --
OBJECT REFCOUNT     -- Yes        --
OBJECT HELP         -- Yes        -- *E
PERSIST             -- Yes        --
PEXPIRE             -- Yes        --
PEXPIREAT           -- Yes        --
PFADD               -- Yes        --
PFCOUNT             -- Yes        -- *2
PFMERGE             -- Yes        -- *2
PING                -- Yes        --
PSETEX              -- Yes        --
PSUBSCRIBE          -- No         --
PTTL                -- Yes        --
PUBLISH             -- No         --
PUBSUB              -- No         --
PUNSUBSCRIBE        -- No         --
QUIT                -- Yes        --
RANDOMKEY           -- Yes        -- *1
READONLY            -- No         --
READWRITE           -- No         --
RENAME              -- Yes        -- *2
RENAMENX            -- Yes        -- *2
RESTORE             -- Yes        --
ROLE                -- Yes        -- *K
RPOP                -- Yes        --
RPOPLPUSH           -- Yes        -- *2
RPUSH               -- Yes        --
RPUSHX              -- Yes        --
SADD                -- Yes        --
SAVE                -- Yes        -- *6
SCAN                -- Yes        -- *A *B
SCARD               -- Yes        --
SCRIPT DEBUG        -- No         --
SCRIPT EXISTS       -- Yes        -- *D
SCRIPT FLUSH        -- Yes        -- *6
SCRIPT KILL         -- No         -- *H
SCRIPT LOAD         -- Yes        -- *6
SDIFF               -- Yes        -- *2
SDIFFSTORE          -- Yes        -- *2
SELECT              -- No         --
SET                 -- Yes        --
SETBIT              -- Yes        --
SETEX               -- Yes        --
SETNX               -- Yes        --
SETRANGE            -- Yes        --
SHUTDOWN            -- Yes        -- *6
SINTER              -- Yes        -- *2
SINTERSTORE         -- Yes        -- *2
SISMEMBER           -- Yes        --
SLAVEOF             -- No         -- *H
SLOWLOG             -- Yes        -- *5
SMEMBERS            -- Yes        --
SMOVE               -- Yes        -- *2
SORT                -- Yes        -- *2 *3
SPOP                -- Yes        --
SRANDMEMBER         -- Yes        --
SREM                -- Yes        --
SSCAN               -- Yes        --
STRLEN              -- Yes        --
SUBSCRIBE           -- No         --
SUNION              -- Yes        -- *2
SUNIONSTORE         -- Yes        -- *2
SWAPDB              -- No         --
SYNC                -- No         --
TIME                -- Yes        -- *6
TOUCH               -- Yes        --
TTL                 -- Yes        --
TYPE                -- Yes        --
UNLINK              -- Yes        -- *4
UNSUBSCRIBE         -- No         --
UNWATCH             -- No         --
WAIT                -- No         --
WATCH               -- No         --
XACK                -- Yes        --
XADD                -- Yes        --
XCLAIM              -- Yes        --
XDEL                -- Yes        --
XGROUP              -- Yes        --
XINFO               -- Yes        --
XLEN                -- Yes        --
XPENDING            -- Yes        --
XRANGE              -- Yes        --
XREAD               -- Yes        -- *L
XREADGROUP          -- Yes        -- *L
XREVRANGE           -- Yes        --
XTRIM               -- Yes        --
ZADD                -- Yes        --
ZCARD               -- Yes        --
ZCOUNT              -- Yes        --
ZINCRBY             -- Yes        --
ZINTERSTORE         -- Yes        -- *2
ZLEXCOUNT           -- Yes        --
ZPOPMAX             -- Yes        --
ZPOPMIN             -- Yes        --
ZRANGE              -- Yes        --
ZRANGEBYLEX         -- Yes        --
ZRANGEBYSCORE       -- Yes        --
ZRANK               -- Yes        --
ZREM                -- Yes        --
ZREMRANGEBYLEX      -- Yes        --
ZREMRANGEBYRANK     -- Yes        --
ZREMRANGEBYSCORE    -- Yes        --
ZREVRANGE           -- Yes        --
ZREVRANGEBYLEX      -- Yes        --
ZREVRANGEBYSCORE    -- Yes        --
ZREVRANK            -- Yes        --
ZSCAN               -- Yes        --
ZSCORE              -- Yes        --
ZUNIONSTORE         -- Yes        -- *2

Notes:
*0 -- Scripts that affect no keys will run on a random backend.
*1 -- Distribution of random keys may not be exactly uniform. RANDOMKEY is
      implemented by choosing a random backend and sending RANDOMKEY to it, so
      if backend A has more keys than backend B, the probability of returning
      each key from backend B is higher.
*2 -- The affected keys must all be on the same backend. If they aren't, the
      command fails with PROXYERROR.
*3 -- The proxy does not check that all the affected keys are on the same
      backend; the application has to do this itself. (This is because complex
      patterns may be given in e.g. the GET clause.)
*4 -- These commands are atomic only on each backend; they are not atomic across
      all backends.
*5 -- The proxy's response format is different than that of Redis - the proxy
      returns a multi response with one field per backend. If this isn't what
      you want, you can use FORWARD to interact with a single backend at once.
*6 -- These commands are forwarded to all backends.
*7 -- Since the proxy doesn't support multiple redis DBs, FLUSHDB is essentially
      equivalent to FLUSHALL.
*8 -- INFO syntax is different from what redis-server expects. These are the
      valid forms of the INFO command in redis-shatter:
      - INFO - return proxy information
      - INFO BACKEND <backend-name> - return proxy stats for a backend
      - INFO <backend-name> [section] - send INFO to a specific backend and
        return its response verbatim
      For the first two forms, the response format is different as well, but
      should be mostly self-explanatory. The third form is equivalent to the
      command FORWARD <backend-name> INFO [section].
*9 -- May be slow and consume lots of memory if run on large datasets. It's
      almost always better to use SCAN instead.
*A -- Keys may be considered or returned multiple times or be inaccessible
      through the proxy if they exist on the wrong backend. For example, if 'x'
      belongs on backend 3 but exists on backends 5 and 3, then 'x' will appear
      twice in the KEYS or SCAN results, or will be double-counted when running
      DBSIZE. If 'x' exists on backend 5 and not 3, then 'x' will appear once in
      the KEYS/SCAN results but 'GET x' will return nil.
*B -- SCAN over the entire keyspace is implemented by scanning on each backend
      in turn. The proxy uses the highest-order bits of the cursor to keep track
      of which backend is currently being scanned. If the backend returns a
      cursor that has any of these bits set, the scan will fail. Most practical
      setups shouldn't run into this limit.
*C -- The returned fields are different from redis-server. They are:
      - name: the client's name (this includes the host:port string).
      - cmdrecv: number of commands received from this client.
      - rspsent: number of responses sent to this client.
      - rspchain: response chain length (the number of responses that haven't
        been sent to the client yet because there isn't enough information to
        generate the response - some backends haven't replied yet).
*D -- 1 will be returned for a script only if it exists on all backends - if it
      is missing on one or more backends, 0 is returned.
*E -- These commands are implemented by forwarding them to a random backend. If
      the backends are not configured identically (i.e. some have rename-command
      directives in their configs and some don't) then the results may differ
      between subsequent calls.
*F -- Names pertain only to the connection between the client and the proxy.
*G -- EVALSHA is more likely to fail in a sharded environment, since the script
      needs to be loaded into all the backends' script caches for it to work on
      arbitrary keys. Although EVAL implicitly loads the script into the script
      cache, EVAL is only forwarded to one backend at a time. If you plan to run
      a script many times on different keys, use SCRIPT LOAD first to load the
      script on all backends at once.
*H -- Note that any unsupported commands can still be run on individual backends
      by using the FORWARD command, but be careful when doing this. See below.
*I -- Which command referred you to this note?
*J -- If any backend returns an error, then MIGRATE returns a multi response
      containing the responses from all backends. If no backend returns an
      error, then MIGRATE returns NOKEY if all backends returned NOKEY, and OK
      otherwise.
*K -- This command returns a multi response with two fields. The first field is
      the string "proxy"; the second field is a multi response containing the
      names of all of the backends.
*L -- Blocking reads are not supported (the BLOCK argument to these commands
      must not be given).


Administration
--------------

redis-shatter implements many administrative commands that are omitted in other
similar proxies. See the commands table above for details.

redis-shatter also implements some administrative commands that aren't part of
the official Redis protocol. These commands are:

BACKEND key [key ...]
  Returns the name of the backend on which the given key belongs, as a data
  response. If multiple keys are given, returns a multi response with one data
  element per key.

BACKENDNUM key [key ...]
  Returns the number of the backend on which the given key belongs, as an
  integer response. If multiple keys are given, returns a multi response with
  one integer element per key.

BACKENDS
  Returns a list of all backends. The items are formatted as "host:port@name".

FORWARD backend-name command [args...]
  Forwards the given command directly to the given backend and return its
  response verbatim. You can use this in lieu of connecting directly to the
  backend. This command shares the backend connections with all other clients,
  so forwarding commands that affect connection state like MULTI or SELECT will
  cause misbehavior.

FORWARD "" command [args...]
  Forwards the given command to all backends and returns a multi response
  containing the backends' responses. As noted above, don't forward commands
  that affect connection state.

PRINTSTATE
  Prints the proxy's internal state to stderr.