Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Keyspace Events in Redis Cluster get lost [DATAREDIS-534] #1111

Open
spring-projects-issues opened this issue Jul 18, 2016 · 1 comment
Open

Keyspace Events in Redis Cluster get lost [DATAREDIS-534] #1111

spring-projects-issues opened this issue Jul 18, 2016 · 1 comment
Assignees
@christophstrobl
Labels
in: core Issues in core support type: bug A general bug

Comments

@spring-projects-issues
Copy link

Mark Paluch opened DATAREDIS-534 and commented

Keyspace notifications in Redis Cluster are published only to the local node (see redis/redis#2541. Keyspace notifications are not broadcasted to all nodes).

Spring Data Redis subscribes only to one (random) node and so the expiry event may be received but that depends on which node the subscription resides and on which cluster node a key expired.

The recommended workaround is to:

  1. Subscribe to all master nodes
  2. Periodically check for topology changes
  3. Handle (merge) events on the client

Affects: 1.7.2 (Hopper SR2)

Issue Links:

  • DATAREDIS-1123 Clustered Redis keyspace notifications are only received from one node in a cluster
    ("is duplicated by")
  • DATAREDIS-1213 Spring-data-redis - Not subscribing to all master nodes
    ("is duplicated by")

2 votes, 3 watchers
@spring-projects-issues spring-projects-issues added type: bug A general bug in: core Issues in core support labels Dec 30, 2020
@marcusdacoregio marcusdacoregio mentioned this issue Jan 31, 2023
Closed
@kinsersh
Copy link
Contributor

kinsersh commented Mar 13, 2024 •
edited

I'm seeing this same issue as reported with spring-projects/spring-session#2230. In my use case, it negatively affects Spring Authorization Server when using SpringSessionBackedSessionRegistry with a sharded redis where the authorization server's token endpoint ends up reading every session including the expired ones in the principal index. The expired sessions stack up which linearly increases the token endpoint latency.

kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 19, 2024
@kinsersh
Remove expired session from principal index
4fa09f4 
Despite RedisIndexedSessionRepository's efforts to automatically remove expired sessions from its principal index, it is possible for expired sessions to remain in that index due to misses from at least these causes:
- application failure due to no event redelivery (see https://docs.spring.io/spring-data/redis/reference/redis/redis-repositories/expirations.html)
  > Redis Pub/Sub messages are not persistent. If a key expires while the application is down, the expiry event is not processed, which may lead to secondary indexes containing references to the expired object.
- application only gets events from one redis node in a cluster (see spring-projects/spring-data-redis#1111)

The findByIndexNameAndIndexValue method in RedisIndexedSessionRepository reads every session in the principal index including the expired ones, only returning the live ones. As expired sessions stack up, it slows down the findByIndexNameAndIndexValue method. Even though RedisIndexedSessionRepository encounters expired sessions, it doesn't take the opportunity to delete them from the principal index. The effect is that expired sessions are read over and over, never getting cleaned up. This was observed negatively affecting Spring Authorization Server's token endpoint due to its reliance on findByIndexNameAndIndexValue via SessionRegistry's getAllSessions method (see https://github.com/spring-projects/spring-authorization-server/blob/29a8321ab2d65f2f479311c28dd72b3809353722/oauth2-authorization-server/src/main/java/org/springframework/security/oauth2/server/authorization/authentication/OAuth2AuthorizationCodeAuthenticationProvider.java#L323).

 This commit improves RedisIndexedSessionRepository to delete the expired session from the principal index when findByIndexNameAndIndexValue sees that a session is expired. This mitigates the impact of expired sessions in that they will only be read once by findByIndexNameAndIndexValue before being removed out of the index.

 Mitigates fh-2230.
kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 19, 2024
@kinsersh
Remove expired session from principal index
2179dc8 
Despite RedisIndexedSessionRepository's efforts to automatically remove expired sessions from its principal index, it is possible for expired sessions to remain in that index due to misses from at least these causes:
- application failure due to no event redelivery (see https://docs.spring.io/spring-data/redis/reference/redis/redis-repositories/expirations.html)
  > Redis Pub/Sub messages are not persistent. If a key expires while the application is down, the expiry event is not processed, which may lead to secondary indexes containing references to the expired object.
- application only gets events from one redis node in a cluster (see spring-projects/spring-data-redis#1111)

The findByIndexNameAndIndexValue method in RedisIndexedSessionRepository reads every session in the principal index including the expired ones, only returning the live ones. As expired sessions stack up, it slows down the findByIndexNameAndIndexValue method. Even though RedisIndexedSessionRepository encounters expired sessions, it doesn't take the opportunity to delete them from the principal index. The effect is that expired sessions are read over and over, never getting cleaned up. This was observed negatively affecting Spring Authorization Server's token endpoint due to its reliance on findByIndexNameAndIndexValue via SessionRegistry's getAllSessions method (see https://github.com/spring-projects/spring-authorization-server/blob/29a8321ab2d65f2f479311c28dd72b3809353722/oauth2-authorization-server/src/main/java/org/springframework/security/oauth2/server/authorization/authentication/OAuth2AuthorizationCodeAuthenticationProvider.java#L323).

 This commit improves RedisIndexedSessionRepository to delete the expired session from the principal index when findByIndexNameAndIndexValue sees that a session is expired. This mitigates the impact of expired sessions in that they will only be read once by findByIndexNameAndIndexValue before being removed out of the index.

 Mitigates spring-projects#2230.
@kinsersh kinsersh mentioned this issue Mar 19, 2024
Closed
kinsersh added a commit to kinsersh/spring-data-redis that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions Redis Cluster keyspace events
8e1dad8 
Due to spring-projects#1111, applications can suffer from missing important Keyspace events.

This adds a caution to the docs to help others avoid running into this problem.
kinsersh added a commit to kinsersh/spring-data-redis that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions about Redis Cluster keyspace events.
78c30eb 
Due to spring-projects#1111, applications can suffer from missing important Keyspace events as described at spring-projects/spring-session#2230.

This adds a caution to the docs to help others avoid running into this problem.
@kinsersh kinsersh mentioned this issue Mar 25, 2024
Closed
4 tasks
kinsersh added a commit to kinsersh/spring-data-redis that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions about Redis Cluster keyspace events.
893a2d0 
Due to spring-projects#1111, applications can suffer from missing important Keyspace events as described at spring-projects/spring-session#2230.

This adds a caution to the docs to help others avoid running into this problem.
kinsersh added a commit to kinsersh/spring-data-redis that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions about Redis Cluster keyspace events.
1daa85b 
Due to spring-projects#1111, applications can suffer from missing important Keyspace events as described at spring-projects/spring-session#2230.

This adds a caution to the docs to help others avoid running into this problem.
kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions about index repo with Redis Cluster
1355540 
Due to spring-projects/spring-data-redis#1111, applications can suffer from missing important Keyspace events as described at spring-projects#2230.

This adds a caution to the docs to help others avoid running into this problem and suggests an alternative.
kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions about index repo with Redis Cluster
9dd178a 
Due to spring-projects/spring-data-redis#1111, applications can suffer from missing important Keyspace events as described at spring-projects#2230.

This adds a caution to the docs to help others avoid running into this problem and suggests an alternative.

Mitigates spring-projects#2230.
kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 25, 2024
@kinsersh
Doc cautions about index repo with Redis Cluster
7f30834 
Due to spring-projects/spring-data-redis#1111, applications can suffer from missing important Keyspace events as described at spring-projects#2230.

This adds a caution to the docs to help others avoid running into this problem and suggests an alternative.

Mitigates spring-projects#2230.
@kinsersh kinsersh mentioned this issue Mar 25, 2024
Closed
kinsersh added a commit to kinsersh/spring-data-redis that referenced this issue Mar 26, 2024
@kinsersh
Doc cautions about Redis Cluster keyspace events.
41cff90 
Due to spring-projects#1111, applications can suffer from missing important Keyspace events as described at spring-projects/spring-session#2230.

This adds a caution to the docs to help others avoid running into this problem.
kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 26, 2024
@kinsersh
Doc cautions about index repo with Redis Cluster
807227d 
Due to spring-projects/spring-data-redis#1111, applications can suffer from missing important Keyspace events as described at spring-projects#2230.

This adds a caution to the docs to help others avoid running into this problem and suggests an alternative.

Mitigates spring-projects#2230.
kinsersh added a commit to kinsersh/spring-session that referenced this issue Mar 26, 2024
@kinsersh
Remove expired session from principal index
7196eca 
Despite RedisIndexedSessionRepository's efforts to automatically remove expired sessions from its principal index, it is possible for expired sessions to remain in that index due to misses from at least these causes:
- application failure due to no event redelivery (see https://docs.spring.io/spring-data/redis/reference/redis/redis-repositories/expirations.html)
  > Redis Pub/Sub messages are not persistent. If a key expires while the application is down, the expiry event is not processed, which may lead to secondary indexes containing references to the expired object.
- application only gets events from one redis node in a cluster (see spring-projects/spring-data-redis#1111)

The findByIndexNameAndIndexValue method in RedisIndexedSessionRepository reads every session in the principal index including the expired ones, only returning the live ones. As expired sessions stack up, it slows down the findByIndexNameAndIndexValue method. Even though RedisIndexedSessionRepository encounters expired sessions, it doesn't take the opportunity to delete them from the principal index. The effect is that expired sessions are read over and over, never getting cleaned up. This was observed negatively affecting Spring Authorization Server's token endpoint due to its reliance on findByIndexNameAndIndexValue via SessionRegistry's getAllSessions method (see https://github.com/spring-projects/spring-authorization-server/blob/29a8321ab2d65f2f479311c28dd72b3809353722/oauth2-authorization-server/src/main/java/org/springframework/security/oauth2/server/authorization/authentication/OAuth2AuthorizationCodeAuthenticationProvider.java#L323).

This commit improves RedisIndexedSessionRepository to delete the expired session from the principal index when findByIndexNameAndIndexValue sees that a session is expired. This mitigates the impact of expired sessions in that they will only be read once by findByIndexNameAndIndexValue before being removed out of the index.

Mitigates spring-projects#2230.
kinsersh added a commit to kinsersh/spring-data-redis that referenced this issue Mar 26, 2024
@kinsersh
Doc cautions about Redis Cluster keyspace events.
daef841 
Applications can suffer from missing important Keyspace events as described at spring-projects/spring-session#2230.

This adds a caution to the docs to help others avoid running into this problem.

Related ticket spring-projects#1111
marcusdacoregio pushed a commit to spring-projects/spring-session that referenced this issue May 13, 2024
@kinsersh @marcusdacoregio
Doc cautions about index repo with Redis Cluster
8b485b6 
Due to spring-projects/spring-data-redis#1111, applications can suffer from missing important Keyspace events as described at #2230.

This adds a caution to the docs to help others avoid running into this problem and suggests an alternative.

Mitigates #2230.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@christophstrobl christophstrobl

Labels
in: core Issues in core support type: bug A general bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kinsersh @christophstrobl @spring-projects-issues