Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RedisCommandExecutionException: MOVED error when used cluster configuration endpoint [DATAREDIS-898] #1470

Closed
spring-projects-issues opened this issue Nov 27, 2018 · 7 comments

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Nov 27, 2018

Pankaj Dubey opened DATAREDIS-898 and commented

I'm using AWS ElastiCache (Redis) in cluster mode. I'm having two implementations to connect to ElastiCache. One of the implementations is directly using the native Lettuce driver and other using Spring data with Lettuce as underneath driver. AWS ElastiCache has Cluster configuration endpoint. I want to use this endpoint to connect to ElastiCache. I'm able to successfully connect to ElastiCache using cluster endpoint with native Lettuce driver implementation but getting below error when using spring data with cluster endpoint

Note: It works when I use RedisClusterConfiguration with nodes endpoint but not with Cluster endpoint

Spring data: 1.8.9-RELEASE (Using a higher version of Spring is not an option) 

Lettuce: 4.5.0-FINAL

Caused by: org.springframework.data.redis.RedisSystemException: 
Error in execution; nested exception is com.lambdaworks.redis.RedisCommandExecutionException: 
MOVED 12894 cache---.usw2.cache.amazonaws.com:6379 at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:50) 
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:48) 
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41) 
 
@Bean
public LettuceConnectionFactory redisConnectionFactory() { 
   LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(); 
   lettuceConnectionFactory.setHostName(<cluster_endpoint>); 
   lettuceConnectionFactory.setPort(); 
   lettuceConnectionFactory.setUseSsl(Boolean.valueOf(useSsl)); 
   return lettuceConnectionFactory; }
@Bean
public RedisTemplate<String, Object> redisTemplate() {
    RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
    redisTemplate.setConnectionFactory(redisConnectionFactory());
    return redisTemplate;
}

Affects: 1.8.16 (Ingalls SR16)

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 27, 2018

Mark Paluch commented

You need to configure  LettuceConnectionFactory with RedisClusterConfiguration so that the connection factory is aware it should use Redis Cluster mode:

 ```java
RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
clusterConfiguration.clusterNode("host", port);
new LettuceConnectionFactory(clusterConfiguration);


Besides that, there isn't anything else we could provide

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 27, 2018

Pankaj Dubey commented

I'm using RedisClusterConfiguration in which I need to pass node endpoints. But I want to use a cluster configuration endpoint (behind which all the nodes are available in elasticache).

@Bean
public RedisClusterConfiguration redisClusterConfiguration() {
    RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration()
            .clusterNode("redis-cluster----0001-001.redis-cluster---.usw2.cache.amazonaws.com",6379)
            .clusterNode("redis-cluster----0001-002.redis-cluster---.usw2.cache.amazonaws.com",6379)
            .clusterNode("redis-cluster----0001-003.redis-cluster---.usw2.cache.amazonaws.com",6379)
            .clusterNode("redis-cluster----0001-004.redis-cluster---.usw2.cache.amazonaws.com",6379);
    return clusterConfiguration;
}

As per my understanding, if one of the node endpoints which is added to RedisClusterConfiguration is correct, native Redis driver (in this case lettuce) will auto-discover all other nodes and establish the connection. But what if all the nodes are replaced with new set of nodes? In the case of cluster endpoint, if all the cluster nodes are replaced with a new set of nodes, cluster endpoint wouldn't change (we might have to refresh the cluster topology though).

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 27, 2018

Mark Paluch commented

So what happens if you use RedisClusterConfiguration and only add the single cluster endpoint?

but getting below error when using spring data with cluster endpoint

The explanation isn't really helpful. Can you provide more details to us?

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 28, 2018

Pankaj Dubey commented

Thanks Mark, now we are able to connect to AWS Elasticache using cluster configuration end point with below code.

RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
clusterConfiguration.clusterNode("host", port);
new LettuceConnectionFactory(clusterConfiguration);

Follow up questions:

  1. How will the connections get refreshed in case of automatic failover?
    1.1 Will setting lettuceConnectionFactory.setValidateConnection to true take care of resetting any invalid connections due to above events?
  2. What's the best way to adapt to topology changes due to scale up/down events? Should we need to explicitly specify topology refresh strategy or Spring Data Redis will adapt automatically?
  3. Is it right to assume Spring Data Redis/Lettuce internally figure out the shards and corresponding key slots to intelligently route set/get requests to right master/slave nodes?

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 28, 2018

Mark Paluch commented

To your questions (these are Lettuce-specific):

  1. You can provide [ClusterTopologyRefreshOptions](https://lettuce.io/core/release/api/io/lettuce/core/cluster/ClusterTopologyRefreshOptions.html) via [{{ClusterClientOptions](https://lettuce.io/core/release/api/io/lettuce/core/cluster/ClusterClientOptions.html)}} and LettuceClientConfiguration
  2. No, setValidateConnection is only intended for standalone connections. If your Redis Cluster is down, then connection validation won't really help. If you reconfigure your cluster, Lettuce takes care of this itself if topology refresh is enabled.
  3. Enable Adaptive and Periodic Refresh. Ideally, you provide an endpoint in your application to get hold of RedisClusterClient and call reloadPartitions() during such an event. Lettuce's reference docs will provide you with additional details.
  4. Yes, Lettuce does all the command routing

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 30, 2018

Pankaj Dubey commented

Thanks Mark.

It seems LettuceClientConfiguration is added in Spring data 2.x release. Since we're using Spring data: 1.8.9-RELEASE & Lettuce: 4.5.0-FINAL, Is there any workaround to refresh the cluster topology. Upgrading the Spring data is not an option for us

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Nov 30, 2018

Mark Paluch commented

You can do the same for Spring Data Redis 1.8.x but this requires a bit of reflection as you need to extract the RedisClusterClient from LettuceConnectionFactory to configure refresh options

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants