Redis Sharding on Haskell, strict version
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
LICENSE first commit Jan 8, 2013
MyForkManager.hs comments in utf8 May 20, 2013
MyListBuf.hs Pipeline acceleration Jan 24, 2013
README use ghc 7.8 Jun 11, 2014
RedisParser.hs comments in utf8 May 20, 2013
RedisSharding.cabal cabal Jan 25, 2013
RedisSharding.hs INCRBYFLOAT added Jan 23, 2014
redis_sharding.hs comments in utf8 May 20, 2013


Redis Sharding


Redis Sharding is a multiplexed proxy-server, designed to work with the database divided to several servers.
It's a temporary substitution of Redis Cluster ( that is under development.

Redis Sharding is used for horizontal Redis database scaling (with connecting of additional servers) as long as load distribution between the cores on the multiprocessor servers (as Redis server is single-threaded, several copies of the server can be run, one for each free core).

                              /- Redis (node 1)
 Client 1 ---                /-- Redis (node 2)
              Redis Sharding --- Redis (node 3)
 Client 2 ---                \-- Redis (node 4)
                              \- Redis (node 5)

Sharding is done based on the CRC32 checksum of a key or key tag ("key{key_tag}").

The majority of the commands are supported except the key renaming commands, 
commands of working with sets, transactions, subscription and, of course, server configure. 

Redis Sharding with fully support MSET, MSETNX, MGET, DEL commands.

For BLPOP and BRPOP commands all the keys should be on one node, that is done with the key tag.
Also for this commands Redis timeout 0 is recommended. 

For SUBSCRIBE, UNSUBSCRIBE and PUBLISH commands Redis timeout 0 is recommended.


Strict Haskell version
Lazy   Haskell version
Perl version 

Strict Haskell version is the best of them all!

Nota bene

To achieve the best efficiency while sharding data it is especially important to use pipelining by the client.
The importance increases according to the number of node for multikeys commands (multi-nodes commands, if more precisely).
And it increases twice only in comparison with the usual redis server. :-)

Some benchmarks


ATTENTION! I would recommend using ghc7.8 which includes the new IO manager. 
This productivity in 1.8 - 1.9 times (

	ghc -threaded -rtsopts -O2 -feager-blackholing --make redis_sharding.hs
	cabal configure && cabal build


	./redis_sharding --nodes=,,...

Others parameters:

	--timeout=300 (0 - disable timeout, seconds)

To use x CPU core run as

	./redis_sharding --nodes=,,... +RTS -Nx

ATTENTION! +RTS -Nx must be after of all others parameters. To use 4 CPU core: -N4

	./redis_sharding --nodes=,,... +RTS -N -A10M -qa

-Nx - Use x simultaneous threads when running the program.
-A  - Set the allocation area size used by the garbage collector
-qa - Use the OS's affinity facilities to try to pin OS threads to CPU cores.
      This is an experimental feature, and may or may not be useful. 


Cluster configuration change on the fly is not supported. We are looking forward for Redis Cluster release. 

To copy data to a new cluster use a new utility

For example, there is a cluster consisting of two servers: A1 and A2. It is served with redis_sharding, that is launched as

 perl --nodes=A1,A2

We want to copy data from the base 9 to a new cluster of 5 servers: B1, B2, B3, B4 and B5.

To do this we need to stop and run for each server of the cluster: 

 perl --db=9 --from=A1 --nodes=B1,B2,B3,B4,B5
 perl --db=9 --from=A2 --nodes=B1,B2,B3,B4,B5

After copying have been done, run for a new cluster: 

perl --nodes=B1,B2,B3,B4,B5

If a flag --flushdb is set, the FLUSHDB command is sent to all cluster nodes before data copying.