How to rehash and devide an exist redis into several redis instances?

[highkay]

We are running a real time log collect and simple anaylze service with one redis instance. While it have been work fine about half a year and the memory consumption come to about 8g, our server is not strong enough to serve such a big guy. Now we want to devide/cluster more redis instances for the service, here comes the problem, I am really exciting finding the cool twemproxy, but according to the doc I found nothing about rehash an exist redis instance like "redis-sharding-hs". What should I do in the situation?

[manjuraj]

You have to write separate client app, that reads all the keys from existing 8 g redis server and write those keys to a cluster of backend redis server through twemproxy

let me know if this works

[charsyam]

hi @highkay , There is a trick.
1] run new redis server
2] and run "slaveof original-ip original-port" on new node
3] run twemproxy
4] run "slaveof no one" on new node

But, this solution is somewhat dangerous.
because one of them fails, It can show old version data which it has.

[manjuraj]

@charsyam I like your idea

[highkay]

hi @charsyam If I am not misunderstand your idea, after 1,2 and 4, it made a full copy of my original node on the new node, while the two full copies are not rehashed at all!

[charsyam]

@highkay hi. so I said it is a kind of trick.

first of all, twemproxy will hash your keys to distribute your queries.

so if you make a full copy of your original node.

It looks like rehashed two nodes.

It is because twemproxy will distribute your queries to 2 nodes.

[highkay]

hi @manjuraj I am sure the twemproxy would be convenience to run on a fresh redis cluster, but in my situation, it seems I would have to write my resharding programme. I wonder if there could be a solution like https://github.com/kni/redis-sharding-hs.

[highkay]

@charsyam First of all, I am sorry for maybe I am not clear my situation. Now we are not using twemproxy, only direct redis connections in our code. So the keys on the original node are not hashed at all. If I made the slave fresh new node fully copy the keys/data, it helped nothing.