Too many Cluster redirections

[yiwiz-sai]

when I tested in 127.0.0.1 ,everything is ok.
but when I tested in remote machine, it got this error every time.

Traceback (most recent call last):
  File "test_redis.py", line 22, in <module>
    main()
  File "test_redis.py", line 18, in main
    print rediscluster.set('foo', 'a')
  File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 1055, in set
    return self.execute_command('SET', *pieces)
  File "/usr/local/lib/python2.7/dist-packages/rediscluster/utils.py", line 82, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rediscluster/client.py", line 307, in execute_command
    raise RedisClusterException("Too many Cluster redirections")
rediscluster.exceptions.RedisClusterException: Too many Cluster redirections

my test code:

# -*- coding: utf-8 -*
import os,sys,time,traceback

import redis
from rediscluster import RedisCluster

def main():
    serverip='10.5.79.60'
    startup_nodes=[{"host": serverip,"port": i} for i in xrange(7000, 7006)]
    try:
        rediscluster = RedisCluster(startup_nodes=startup_nodes)
    except Exception, err:
        print 'failed to connect cluster'
        sys.exit(0)

    for i in xrange(1000):
        print rediscluster.set('foo', 'a')

if __name__=='__main__':
    main()

I used pip install, that version could not import "StrictRedisCluster"
from rediscluster import StrictRedisCluster

[Grokzen]

Hi. This is abit tricky to debug remotely but i can give it a try :]

When i see this usually there is a problem to connect to the cluster because when it sends the command it tries to connect to what it thinks is the node it should connect to but for some reason it can't after trying a number of times.

I know i have seen this for example when running the cluster inside some virtual machine or docker container because there is a known bug there because when a client queries redis for its cluster nodes it will return the internal IP address of each node and not a valid external IP that can be accessed from outside of the VM/container.

I would start by looking what your redis-cluster returnes when queried for the cluster information.

I get the following output and can you paste your output?

$ redis-cli -p 7001
127.0.0.1:7001> cluster nodes
7d19fd07769113015011ffca8611a8aa3ef47391 127.0.0.1:7000 master - 0 1430251094775 1 connected 0-5460
aa479daca239c1b4b3122bbeddcc779ac1060a69 127.0.0.1:7002 master - 0 1430251096780 3 connected 10923-16383
18a10efad473e145a08c5959212e737343b705b7 127.0.0.1:7005 slave aa479daca239c1b4b3122bbeddcc779ac1060a69 0 1430251095275 6 connected
53dff86ee714f77fca3656aff2c4e4a915f62774 127.0.0.1:7004 slave bfc2cd20c4a8f5c5b203901afeaa1f5b5f4442c4 0 1430251096278 5 connected
bfc2cd20c4a8f5c5b203901afeaa1f5b5f4442c4 127.0.0.1:7001 myself,master - 0 0 2 connected 5461-10922
bfd520ca2c64b1e1925652ae21aaa94612924fbe 127.0.0.1:7003 slave 7d19fd07769113015011ffca8611a8aa3ef47391 0 1430251095776 4 connected
127.0.0.1:7001

I would expect that you will get something similar and that will explain why it will not work because this lib will parse that your nodes should exists on 127.0.0.1:7000 for example and when you try to do a SET on that cluster slot it will try to connect to it but fails. If this is not the case we have to dig deeper inside the code :]

[yiwiz-sai]

yes, it just got this(I didn't use slave)

$ redis-cli  -p 7000 -h 10.5.79.60
10.5.79.60:7000> cluster nodes
011b424c7c90b1499db66edaa8c799087f09ba65 127.0.0.1:7005 master - 0 1430317722574 6 connected 13653-16383
c970c8b713f64aa01909f938d08ccdd427e7ca53 127.0.0.1:7004 master - 0 1430317722574 5 connected 10923-13652
3f8f4875c86fe26203044476eb9e9b5f0c8d0ed8 127.0.0.1:7001 master - 0 1430317723075 2 connected 2731-5460
cc1de4362949d60a5e8e10499bb06df4af6393a3 127.0.0.1:7003 master - 0 1430317723578 4 connected 8192-10922
f1124230ae0a85219b85aeee63ea85f392bb7672 127.0.0.1:7000 myself,master - 0 0 1 connected 0-2730
837c74d8329c5fe832eb89a209419765aa6ed26c 127.0.0.1:7002 master - 0 1430317723578 3 connected 5461-8191

I did't run cluster in any vm. how could I solve this problem.
thanks

[Grokzen]

Tbh, i do not know how to solve this really because this is a problem that many people have and it exists inside redis and not inside this lib. One way would be not to use a virtualization layer and use a real server or run locally on the same machine. If you get behind some NAT like interface then redis fails.

There might be a workaround inside the code however but i would not recommend it but if you want you could pass init_slot_cache=False into StrictRedisCluster and then build/set the slots cache yourself and set correct IP and make the cluster work like you want. This might be difficult but if you really need to make it work it might be something you can do.

Another thing that could be interesting to experiment with is to build a cluster and use DNS names when joining the cluster together and see if redis reports back the dns names or if it translates it to the closest IP address like in your case.

[Grokzen]

@pianoboysai Any progress on this? I would like to close this ticket as invalid because this is not a problem in this lib but in redis or how you setup your cluster.

If possible could you try to build your cluster by providing the external IP addresses when you are are running redis-trib.rb. If my theory is correct then redis would internally track the external IP addresses for each node and when you run a cluster client it should get the correct IP addresses and it will work as expected.

[yiwiz-sai]

ok, thanks

2015-05-11 23:19 GMT-07:00 Grok notifications@github.com:

@pianoboysai https://github.com/pianoboysai Any progress on this? I
would like to close this ticket as invalid because this is not a problem in
this lib but in redis or how you setup your cluster.

If possible could you try to build your cluster by providing the external
IP addresses when you are are running redis-trib.rb. If my theory is
correct then redis would internally track the external IP addresses for
each node and when you run a cluster client it should get the correct IP
addresses and it will work as expected.

—
Reply to this email directly or view it on GitHub
#67 (comment)
.

Best Regards,
Jin

[dharmit]

Hi @Grokzen, I faced this issue while testing on localhost.

Traceback (most recent call last):
  File "test_redis.py", line 13, in <module>
    rc.set("foo%d" % i, i)
  File "/home/dharmit/redis/venv/local/lib/python2.7/site-packages/redis/client.py", line 1055, in set
    return self.execute_command('SET', *pieces)
  File "/home/dharmit/redis/venv/local/lib/python2.7/site-packages/rediscluster/utils.py", line 82, in inner
    return func(*args, **kwargs)
  File "/home/dharmit/redis/venv/local/lib/python2.7/site-packages/rediscluster/client.py", line 307, in execute_command
    raise RedisClusterException("Too many Cluster redirections")
rediscluster.exceptions.RedisClusterException: Too many Cluster redirections

I am testing the master-slave setup mentioned in this tutorial.

My code:

from rediscluster import RedisCluster

# import time

startup_nodes = [{"host": "127.0.0.1", "port": "7000"},
                 {"host": "127.0.0.1", "port": "7001"}]

rc = RedisCluster(startup_nodes=startup_nodes, decode_responses=True)

# last = False

for i in xrange(1000000):
    rc.set("foo%d" % i, i)
    print rc.get("foo%d" % i)
    rc.set("__last__", i)

    # time.sleep(1)

If this helps:

./redis-cli -p 7000 cluster nodes
9f2f86c3824aa0cd3fcdeabde4fdd367d5640239 127.0.0.1:7004 slave 4c66d2caafc4946f27bae89686e919a1ef5247c3 0 1431605525130 8 connected
90bba532fcb5e3302ec761d5d951e0af47bfb39e 127.0.0.1:7005 slave c5fa4ea3b9162c7ea7d134a64c00b43606387f14 0 1431605524628 6 connected
4c66d2caafc4946f27bae89686e919a1ef5247c3 127.0.0.1:7001 master - 0 1431605526133 8 connected 0-5656 5962-10922 11422-15764
cc9226afbcefcbdc04a33357ba3073e0087394f0 127.0.0.1:7003 slave 0bc03a7b52dfedcb441c07b3647d0e8009d238fe 0 1431605525632 7 connected
c5fa4ea3b9162c7ea7d134a64c00b43606387f14 127.0.0.1:7002 master - 0 1431605526633 3 connected 15765-16383
0bc03a7b52dfedcb441c07b3647d0e8009d238fe 127.0.0.1:7000 myself,master - 0 0 7 connected 5657-5961 10923-11421

[Grokzen]

@dharmit So you are running your nodes on the same machine that you are running your redis cluster? You are not running the cluster inside a VM, Docker container or other machine and your redis-py-cluster client on your local host?

[dharmit]

@Grokzen No VM. No Docker container. Same machine running redis-server and redis-py-cluster client.

[Grokzen]

@dharmit The error is kinda generic and can mean alot of things but usually it means that the client get stuck inside this while loop https://github.com/Grokzen/redis-py-cluster/blob/unstable/rediscluster/client.py#L278 and the most common issue is that the IP address it gets back from the cluster is unreachable or wrong. If you are inside docker for example and you create a cluster, redis usually tracks 127.0.0.1 as the IP for the other nodes and it will work as long as the client is alos running from the inside but when you try to reach it from the outside then you get this kind of error.

My suggestion is that you have to do some digging yourself inside the while loop i linked above. The code should throw a exception and you have to look what one of the three except blocks that gets caught and see what the exception says. I will guess that you get stuck in this one https://github.com/Grokzen/redis-py-cluster/blob/unstable/rediscluster/client.py#L301 but please share what you find in case there is some bug in the code.

Could you state what python version and what version of redis-py-cluster you are using? unstable branch or 0.2.0 release?

[dharmit]

@Grokzen

Python 2.7.9
redis-py-cluster==0.2.0

I was performing sharding using redis-trib.rb reshard 127.0.0.1:7000 while the script was running. Could that be the cause of this issue?

[Grokzen]

Ye that is probably the cause of it but i can't tell what or why you get that error. I have never seen that error happening during resharding. I could try some things on my side and my setup to see if i can replicate the problem but you should really dig into your setup and see what the code is doing.

[Grokzen]

@dharmit Any updates on your issues regarding this? I would really like to close these issues if they are still not applicable or invalid.

[dharmit]

@Grokzen no updates from my side at the moment. Please feel free to close it.