Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClusterAllFailedError: Failed to refresh slots cache. with Cluster mode and TLS enabled #1454

Open
jungRoit opened this issue Nov 5, 2021 · 4 comments

Comments

@jungRoit
Copy link

jungRoit commented Nov 5, 2021

I'm trying to configure an ElastiCache cluster with TLS and Auth configured. I'm getting the following error with trying to connect:

ClusterAllFailedError: Failed to refresh slots cache

Here's my configurations detatils:
ioredis: 4.27.8
ElastiCache: 6.x

Code snippet:
client = new Redis.Cluster([{host: HOST,port:PORT}],{ redisOptions: { password: PASSWORD, tls: {} } });

@anax015obs
Copy link

I have a same problem.. tls: {} or dnsLookup does not work.

@juanviamonte
Copy link

juanviamonte commented Feb 20, 2022

This is an extremely annoying issue tbh, is you search under Issues tab for this repo there is at least 15 issues related to the same ClusterAllFailedError: Failed to refresh slots cache.

I had the same problem i was able to work around but not 100% because from time to time it disconnects and timeout.

  1. If you are using cluster then make sure your ElastiCache Redis Instance Cluster mode is on (you have to make sure this is selected at the moment of creation)

image

  1. Make sure there is AUTH enabled, and TLS, (Encryption at Rest ON, Encryption in Transit ON)
  2. Make sure you can reach the primary endpoint of the Redis Cluster
redis-cli -u redis://PUT_YOUR_AUTH_PASSWORD_HERE@master.your_endpoint.use1.cache.amazonaws.com:6379/0 --tls ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG

^ if you see PONG means your computer or EC2 can reach that Redis endpoint. If you're running from your actual computer (LInux, Mac, Windows, etc) you need to be inside the AWS VPN if you are not inside an AWS VPN from your organization you will not reach the Redis Cluster.
4. Make sure you can connect from an EC2 host in same VPC, or make sure you can connect from a lambda test function before going crazy and putting the code in your project.
5. I had to do this before actually connecting via Node/ioRedis code:

redis-cli -u redis://PUT_YOUR_AUTH_PASSWORD_HERE@master.your_endpoint.use1.cache.amazonaws.com:6379/0 --tls 

^ after connected to redis i did this:
> cluster slots
^ that should print your shard/cluster replicas/endpoints something similar to this:

1) 1) (integer) 0
   2) (integer) 16383
   3) 1) "YOUR_ENDPOINT_1-001.xyz.use1.cache.amazonaws.com"
      2) (integer) 6379
      3) "5d15665843ae2f5528fe01ae9e...."
   4) 1) "YOUR_ENDPOINT33.xyz.use1.cache.amazonaws.com"
      2) (integer) 6379
      3) "3c33fafba73a4270a5387809c861......"
   5) 1) "YOUR_ENDPOINT_2.xyz.use1.cache.amazonaws.com"
      2) (integer) 6379
      3) "daa36bd8b76d8464344...."
  1. This is the code I used: file: redis_test.js
const connectCluster = () => {
  const cluster = new Redis.Cluster(
    [
      {
        host: "clustercfg.your_cluster_address.xyz.use1.cache.amazonaws.com",
        port: 6379,
      },
    ],
    {
      dnsLookup: (address, callback) => callback(null, address),
      redisOptions: {
        tls: true,
        password:
          "YOUR_AUTH_PASSWORD",
      },
    }
  );

  return cluster;
};

const cluster = connectCluster();
cluster.set("name", "test");
cluster.get("name").then(r => {
  console.log(r); // <- you should see "test"
});

Hopefully this help you both @anax015obs and @jungRoit

Also like others mentioned in different Issues use: DEBUG=ioredis:* node redis_test.js in order to see more information about your specific error message.

@jasonmorita
Copy link

This is an extremely annoying issue tbh, is you search under Issues tab for this repo there is at least 15 issues related to the same ClusterAllFailedError: Failed to refresh slots cache.

@juanviamonte THIS WORKED! Thank you for documenting those steps

@SeyyedKhandon
Copy link

SeyyedKhandon commented Jun 3, 2024

This is an extremely annoying issue tbh, is you search under Issues tab for this repo there is at least 15 issues related to the same ClusterAllFailedError: Failed to refresh slots cache.

I had the same problem i was able to work around but not 100% because from time to time it disconnects and timeout.

  1. If you are using cluster then make sure your ElastiCache Redis Instance Cluster mode is on (you have to make sure this is selected at the moment of creation)
image
  1. Make sure there is AUTH enabled, and TLS, (Encryption at Rest ON, Encryption in Transit ON)
  2. Make sure you can reach the primary endpoint of the Redis Cluster
redis-cli -u redis://PUT_YOUR_AUTH_PASSWORD_HERE@master.your_endpoint.use1.cache.amazonaws.com:6379/0 --tls ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG

^ if you see PONG means your computer or EC2 can reach that Redis endpoint. If you're running from your actual computer (LInux, Mac, Windows, etc) you need to be inside the AWS VPN if you are not inside an AWS VPN from your organization you will not reach the Redis Cluster. 4. Make sure you can connect from an EC2 host in same VPC, or make sure you can connect from a lambda test function before going crazy and putting the code in your project. 5. I had to do this before actually connecting via Node/ioRedis code:

redis-cli -u redis://PUT_YOUR_AUTH_PASSWORD_HERE@master.your_endpoint.use1.cache.amazonaws.com:6379/0 --tls 

^ after connected to redis i did this: > cluster slots ^ that should print your shard/cluster replicas/endpoints something similar to this:

1) 1) (integer) 0
   2) (integer) 16383
   3) 1) "YOUR_ENDPOINT_1-001.xyz.use1.cache.amazonaws.com"
      2) (integer) 6379
      3) "5d15665843ae2f5528fe01ae9e...."
   4) 1) "YOUR_ENDPOINT33.xyz.use1.cache.amazonaws.com"
      2) (integer) 6379
      3) "3c33fafba73a4270a5387809c861......"
   5) 1) "YOUR_ENDPOINT_2.xyz.use1.cache.amazonaws.com"
      2) (integer) 6379
      3) "daa36bd8b76d8464344...."
  1. This is the code I used: file: redis_test.js
const connectCluster = () => {
  const cluster = new Redis.Cluster(
    [
      {
        host: "clustercfg.your_cluster_address.xyz.use1.cache.amazonaws.com",
        port: 6379,
      },
    ],
    {
      dnsLookup: (address, callback) => callback(null, address),
      redisOptions: {
        tls: true,
        password:
          "YOUR_AUTH_PASSWORD",
      },
    }
  );

  return cluster;
};

const cluster = connectCluster();
cluster.set("name", "test");
cluster.get("name").then(r => {
  console.log(r); // <- you should see "test"
});

Hopefully this help you both @anax015obs and @jungRoit

Also like others mentioned in different Issues use: DEBUG=ioredis:* node redis_test.js in order to see more information about your specific error message.

Thanks, just one note here, tls is not boolean, so you can pass:

  const redisCluster = new Redis.Cluster([{ host, port }], {
    scaleReads: 'all',
    dnsLookup: (address, callback) => callback(null, address),
    redisOptions: {
      password,
      tls: {},
    },
  })

By the way this didnt fixed this issue for my case.

I ended up for using node-redis:
redis/node-redis#2768

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants