AWS RedisCluster not working / get master node failed

[kstrempel]

We trying to use aredis with a ElasticCache Redis Cluster, but getting the following error.

[2018-01-08 10:21:28 +0000] [9] [ERROR] Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 556, in handle_request
    response = await response
  File "/usr/local/lib/python3.6/site-packages/prometheus_async/aio/_decorators.py", line 42, in decorator
    rv = yield from wrapped(*args, **kw)
  File "/menu_service/menu_response.py", line 75, in get_menu_endpoint
    menu = await get_menu_cache(id)
  File "/menu_service/cache.py", line 21, in get_menu_cache
    result = await redis.get(_cache_key(id))
  File "/usr/local/lib/python3.6/site-packages/aredis/commands/strings.py", line 147, in get
    return await self.execute_command('GET', name)
  File "/usr/local/lib/python3.6/site-packages/aredis/utils.py", line 167, in inner
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/aredis/client.py", line 405, in execute_command
    node = self.connection_pool.get_node_by_slot(slot)
  File "/usr/local/lib/python3.6/site-packages/aredis/pool.py", line 480, in get_node_by_slot
    return self.get_master_node_by_slot(slot)
  File "/usr/local/lib/python3.6/site-packages/aredis/pool.py", line 475, in get_master_node_by_slot
    return self.nodes.slots[slot][0]
KeyError: 232 

Our redis client creation is the following:

startup_nodes = [{"host": REDIS_HOST, "port": REDIS_PORT }]
redis = StrictRedisCluster(startup_nodes=startup_nodes, decode_responses=False, skip_full_coverage_check=True)

We using python 3.6 with sanic 0.7.0.
The setup is working with using the redis-py-cluster==1.3.4.

[NoneGG]

@kstrempel
Sorry to bring you inconvenience.
Since you don't refer the version of aredis, i suppose it to be the newest version(v1.1.1)
This problem seems to be caused by setting skip_full_coverage_check to True(it's surely correct considering the aws env)

I can't reproduce the problem in my dev env so it is hard for me to locate the bug accurately.
I can only see there may be something wrong when node manager of the connection pool is initialized. (connection pool needs to be initialized explicitly or initialized when calling execute_command implicitly, which is one difference between redis-py-cluster and aredis.) And then the slots dict of node manager is not initialized as expected.

If you don't mind, could you help me add log about the redis.connection_pool.initialized and redis.connection_pool.nodes.slots around your code please? Or could you run the script below in your env?

#!/usr/bin/python
# -*- coding: utf-8 -*-

import asyncio
import logging
from aredis import StrictRedisCluster


async def example():
    redis = StrictRedisCluster(startup_nodes=[{'host': '127.0.0.1', 'port': 7001}], decode_responses=False, skip_full_coverage_check=True)
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())
    logging.info(await redis.get('a'))
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())


if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(example())

In a ideal empty cluster(for example, docker redis cluster in my dev env), output should like below

INFO:root:False
INFO:root:dict_keys([])
INFO:root:None
INFO:root:True
INFO:root:dict_keys([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, ..., 16383])

Looking forward for your response.

[kstrempel]

Hello,

sorry for the late anser. I run your small testscript against the AWS redis cluster.

This is the result

INFO:root:False
INFO:root:dict_keys([])
Traceback (most recent call last):
  File "testredis.py", line 21, in <module>
    loop.run_until_complete(example())
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "testredis.py", line 13, in example
    logging.info(await redis.get('a'))
  File "/usr/local/lib/python3.6/site-packages/aredis/commands/strings.py", line 147, in get
    return await self.execute_command('GET', name)
  File "/usr/local/lib/python3.6/site-packages/aredis/utils.py", line 167, in inner
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/aredis/client.py", line 405, in execute_command
    node = self.connection_pool.get_node_by_slot(slot)
  File "/usr/local/lib/python3.6/site-packages/aredis/pool.py", line 480, in get_node_by_slot
    return self.get_master_node_by_slot(slot)
  File "/usr/local/lib/python3.6/site-packages/aredis/pool.py", line 475, in get_master_node_by_slot
    return self.nodes.slots[slot][0]
KeyError: 15495

I had to change the connection part a bit.

async def example():
    redis = StrictRedisCluster(startup_nodes=[{'host': 'redis', 'port': 6379}], decode_responses=False, skip_full_coverage_check=True)
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())
    logging.info(await redis.get('a'))
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())

A AWS you have no 7000, 7001 port for the cluster.

[NoneGG]

@kstrempel
Thanks for your feedback~
Please try code below to see if this problem is caused by not calling connection_pool.initialize or slots not obtained in nodemanager.initialize

#!/usr/bin/python
# -*- coding: utf-8 -*-

import asyncio
import logging
from aredis import StrictRedisCluster


async def example():
    redis = StrictRedisCluster(startup_nodes=[{'host': '127.0.0.1', 'port': 7001}], decode_responses=False, skip_full_coverage_check=True)
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())
    await redis.connection_pool.initialize()
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots[15495])
    logging.info(await redis.get('a'))


if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(example())

[kstrempel]

Here it is.
I had to change the connection part again to 'port': 6397.

INFO:root:False
INFO:root:dict_keys([])
INFO:root:True
Traceback (most recent call last):
  File "test.py", line 22, in <module>
    loop.run_until_complete(example())
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "test.py", line 15, in example
    logging.info(redis.connection_pool.nodes.slots[15495])
KeyError: 15495

BTW I forgot to mention that our current solution is running with the redis-py-cluster==1.3.4 package, with the exact same connection configuration. Perhaps that helps to find the problem.

[NoneGG]

@kstrempel
Thanks.
It seems like that node_manager.initialize get no slots info when called, i will compare with redis-py-cluster code to find the real problem.

[NoneGG]

@kstrempel
Could you help to try the code optimized? I compared this part of code with redis-py-cluster, most logic of them is the same but the process of single node situation.
But i am not sure if it is possible that you have only one node in your aws redis cluster.
you can install the newest version by:
pip install git+https://github.com/NoneGG/aredis
If that still not work, could you show me your cluster slot info please?

[vineetgoel]

I tried that out! It worked!

[NoneGG]

@vineetgoel
Thanks a lot sincerely, that really helps!
@kstrempel
If you don't mind, this issue will be closed, thank you again for you bug report~
If there are still problems, please reopen the issue.