Spring session data redis has issue when AWS Redis cluster upgrade version

[pbluong]

Version: spring-session-data-redis-2.2.5.RELEASE

In my application, I am using Jedis to connect to the Amazon Redis cluster. Everything went well until recently when AWS forced an upgrade of all the Redis server versions from 5.0.3 to 5.0.6.

What we observed is that after the upgrade, the application still works without any restart. However, the org.springframework.data.redis.connection.RedisConnectionCommands#ping took much longer than normal. If we restart the application, the command ping is fast again.

We spent much time to analyze this issue because it was a big issue in our application. In the analysis, we found that the issue comes from the org.springframework.data.redis.connection.jedis.JedisClusterConnection.JedisClusterTopologyProvider#getTopology(..) method when upgrading Redis version (we don't know exactly what Amazon did), but we can see that it created new cluster with new IP addresses and its slaves. The old nodes from the old cluster are cached but they were no longer available. The current logic shuffles the list, then loops over the list and tries to reach the node. So, if the invalid nodes are at the beginning of the list, it will take time to reach the valid node(s). It explains why the ping command is randomly slow & fast. It depends on the shuffle method.

It would be great if the team could have a look and see if it is really a bug?

Thanks and best regards

[jxblum]

First, a bit of feedback...

Spring Data Redis 2.2.x (and in particular 2.2.5-RELEASE) has been out of OSS support since September 30th, 2020 (see support timelines for more details).

Furthermore, it is not even possible to buy "commercial support" (with a valid VMware Spring Runtime license) unless you were already minimally running Spring Data Redis 2.5.x, which will end in August 2023.

Additionally, you are encouraged to always use the latest service release (patch release) of any Spring Data Redis major.minor version, such as the version you are currently on,2.2, which would be 2.2.13-RELEASE (here). There is very little to no risk in using the latest service/patch release of any Spring Data Redis major.minor version, or of any Spring Data module for that matter.

I would even highly recommend moving to Spring Data Redis 2.7 as soon as possible, and even consider moving to Spring Data Redis 3.0 (3.0.5 is the latest service/patch release version in 3.0.x already). The only caveat by moving to SD Redis 3.0 is that it minimally requires Java 17, Jakarta EE 9/10, and is based on Spring Framework 6.0.

Note that, even though Spring Data Redis 2.7 OSS support states it expires this month, May 2023, it is possible that SD Redis 2.7 will get extended until November 2023 to allow users more time to transition from the 2.x Java 8 baseline to the 3.x Java 17 baseline. The reason I say this is, Spring Boot 2.7.x OSS support was originally going to end this May 2023, but has be pushed out to November 2023 (see here). We typically, though not always, follow Spring Boots support timelines.

I am actually requesting clarification on this during tomorrow's (Monday, May 9th, 2023) Spring Data team meeting.

[jxblum]

Having stated the above...

We, the Spring Data team, are actually performing 1 final release of Spring Data Redis 2.7.x (which will be SD Redis 2.7.12), along with 3.0.6 and a 3.1.0 GA, tentatively scheduled on Friday, May 12th, 2023.

Of course, this may change if the Spring Data Redis 2.7.x OSS support timeline is pushed out to November 2023 to match Spring Boot 2.7.x. However, I need to confirm this first.

[jxblum]

Unfortunately, it would appear that this problem, at least inside AWS, would still persist with the latest Spring Data Redis version today, even for Spring Data Redis 3.x and beyond (see here).

The Collections.shuffle(list) call on line 830 was a contribution submitted by another user on November 18th, 2018, related to DATAREDIS-890, with details:

DATAREDIS-890 - Randomize Redis Cluster node order before topology retrieval.

We now shuffle the set of Redis Cluster nodes before retrieving the topology. This change reduces load on the first node in the known nodes set.

Original pull request: #373.

DATAREDIS-890 (from our now retired Spring JIRA Issue Tracker) corresponds to SD Redis GitHub Issue #1427.

It would seem the shuffle was used to solve a different problem (issue with load) having a similar performance degradation hit.

[jxblum]

I was actually wondering whether there is a more intelligent way to "order" the List (here), such that more recent Redis cluster nodes (Connections) could/would be stored in the beginning of the List.

This would require us to know exactly what kind of redis.clients.jedis.ConnectionPool the Jedis driver's, redis.clients.jedis.JedisCluster.getTopology() method returns? Unfortunately, the Javadoc is less than helpful.

The Javadoc for the Jedis ConnectionPool and Connection (Javadoc) classes are equally worthless.

If there were some kind of metadata on the Connection itself that would help with this "smart ordering", then that might be useful and beneficial. But, in the source for the Connection class, as determined by the SD Redis dependency (with version), I see:

package redis.clients.jedis;

import ...;

public class Connection implements Closeable {

  private ConnectionPool memberOf;
  private final JedisSocketFactory socketFactory;
  private Socket socket;
  private RedisOutputStream outputStream;
  private RedisInputStream inputStream;
  private int soTimeout = 0;
  private int infiniteSoTimeout = 0;
  private boolean broken = false;

// ...

Which is not all that helpful in this case.

However, I am curious about 2 things.

1. First, let's assume the Jedis driver, JedisCluster.getTopology() method only returns valid, "active" Connections that produce an immediate response to a Redis command (for example, on ping) from a Redis cluster node. Then perhaps the problem is not with the Connections but with the "cached" topology returned by SD Redis's JedisClusterTopologyProvider, here.

This non-null, "cached" topology will expire (i.e. shouldUseCachedValue), as defined here, only after a set, configurable timeout (i.e. cacheTimeMs), which defaults to 100ms.

100 milliseconds could prove quite detrimental in a really large cluster (100s or 1000s of nodes, depending on the scale of your topology in AWS).

So, you could try this in configuration...

@Configuration
class MySpringDataRedisApplicationConfiguration {

  @Bean
  JedisConnectionFactory customJedisConnectionFactory(JedisClientConfiguration clientConfiguration, 
      RedisClusterConfiguration clusterConfiguration) {

    return new JedisConnectionFactory(clusterConfiguration, clientConfiguration) {

      @Override
      protected ClusterTopologyProvider createTopologyProvider(JedisCluster cluster) {

        // NOTE: I arbitrarily specified a `Duration` in nanoseconds, but you should determine 
        // the correct `Duration` for your application use case, inside AWS, and test it!

        return new new JedisClusterTopologyProvider(cluster, Duration.ofNanos(50));
      }
    };
  }
}

2. Alternatively, you might want to consider changing the SO_TIMEOUT of the actual Sockets used by the Jedis (Cluster) Connections used to issue Redis commands, such as ping. As you can see in the Jedis Connection class definition I showed above (from source), the soTimeout defaults to 0. And, as the Javadoc for java.net.Socket.setSoTimeout(:int) method states:

"A timeout of zero is interpreted as an infinite timeout."

An "infinite timeout" is probably not what you need either, especially if any node becomes unresponsive or is no longer available.

You should be able to accomplish this with the following code:

@Configuration
class MySpringDataRedisApplicationConfiguration {

  @Bean
  JedisClientConfiguration customJedisClientConfiguration() {

    // NOTE: Once again, I chose an arbitrary `Duration`; you must decide what is best for your application!

    // NOTE: I set both `connect` and `read` timeouts, here, incase the node exists, but is unresponsive!

    return JedisClientConfiguration.builder()
      .connectTimeout(Duration.ofNanos(50))
      .readTimeout(Duration.ofNanos(50))
      .build();
  }

  // `JedisConnectionFactory` bean and other bean declarations omitted.
}

DISCLAIMER: I wrote the code from scratch (and memory) so I apologize for any syntax errors in advance. But hopefully you get the idea.

The SD Redis JedisClientConfiguration gets converted into a Jedis driver API JedisClientConfig here.

Please try either approach (or both and perhaps together) I presented above and let me know your results.

I will leave this ticket open for the time being, but I am not convinced that anything inside Spring Data Redis needs to be changed just yet.

Thank you and good luck!

[jxblum]

1 last bit of feedback here, you might want to consider trying the Lettuce Redis (client) driver instead of Jedis if you continue to have problems, even after my suggestions above.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.