Spring Data Redis issue with CRUDRepository

[arianlopezc]

My team and I are using a CRUD repository with Redis to perform some operations on it. The problem is that there is an entry in Redis that gets generated as an index which stores the keys associated to the Redis entries, and this entry never removes entries that were already flushed when TTL reached 0.

Here is an example of the code we use.

@RedisHash("rate")
public class RateRedisEntry implements Serializable {
    @Id
    private String tenantEndpointByBlock; // A HTTP end point
    ...
}

// CRUD repository.
@Repository
public interface RateRepository extends CrudRepository<RateRedisEntry, String> {}

This generates the entry rate in Redis which is the Set object I mentioned before.

When we check the memory usage on it, it just keeps growing all the time until the memory usage reaches 100% from the available in Redis.'

MEMORY USAGE "rate"
(integer) 153034
.
.
MEMORY USAGE "rate"
(integer) 153876
.
.
MEMORY USAGE "rate"
(integer) 163492

Is there a way to prevent this index from being created or for the values stored to be removed once the entries' TTL reaches 0?

Any assistance is appreciated.

[mp911de]

Index cleanup happens on the application side and isn't automatically performed by Redis servers. Spring Data Redis repositories maintain secondary indexes by inserting and removing data from sets. Expirations are detected by listening to keyspace events to perform index cleanup upon expiration. That requires proper keyspace settings (CONFIG SET notify-keyspace-events KEY) and subscribing to keyspace events through Pub/Sub.

If the application is down, then keyspace events can get lost and therefore index maintenance cannot happen.

[arianlopezc]

Index cleanup happens on the application side and isn't automatically performed by Redis servers. Spring Data Redis repositories maintain secondary indexes by inserting and removing data from sets. Expirations are detected by listening to keyspace events to perform index cleanup upon expiration. That requires proper keyspace settings (CONFIG SET notify-keyspace-events KEY) and subscribing to keyspace events through Pub/Sub.

If the application is down, then keyspace events can get lost and therefore index maintenance cannot happen.

What it seems to work is actually this setting:

@EnableRedisRepositories(enableKeyspaceEvents = EnableKeyspaceEvents.ON_STARTUP)

We do set up our TTL for each entry but the Set/Index that is created does not flush the keys away without that option enabled.

Any other recommendations?

[mp911de]

Can you provide a reproducer?

[arianlopezc]

Here is an example on how it should look like so the Index actually flushes away the entry once that the TTL has reached 0.

@Configuration
@EnableRedisRepositories(enableKeyspaceEvents = EnableKeyspaceEvents.ON_STARTUP)
public class ConfigRedis {
@Bean
    @Primary
    public RedisConnectionFactory getLettuceConnectionFactory() {
        ...
        return new LettuceConnectionFactory(...);
    }

@Bean
    public RedisTemplate<String, Object> iasRedisTemplate() {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(getLettuceConnectionFactory());
        return template;
    }
}

//This would be an object to store.
@RedisHash("entry")
public class RedisEntry implements Serializable {
    @Id
    private String id;
    @TimeToLive
    private Integer ttl;
    ...
}

@Repository
public interface RateRepository extends CrudRepository<Object, String> {}

If this decorator:
@EnableRedisRepositories(enableKeyspaceEvents = EnableKeyspaceEvents.ON_STARTUP)
Is not added using that option, then after TTL reaches 0, the key in the entry object stored in Redis as a Set structure created in Redis as an index, it will keep storing everything and not flush it away.

[arianlopezc]

I found a solution to the problem but it wasn't the best for my case. Considering that this was an interesting issue to address on my end, I created an article about it.

Lessons Learned using Spring Data Redis