Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve thread management for Redis streams [DATAREDIS-954] #1503

Closed
spring-projects-issues opened this issue Mar 28, 2019 · 2 comments
Closed
Assignees
Labels
in: core status: declined type: enhancement

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Mar 28, 2019

Akash Kengunte Jayachandra opened DATAREDIS-954 and commented

Hi,

We at Intuit are working on a project and looking to use Redis Streams. I found that Spring Data Redis provides good support for Redis Streams in it's new upcoming version of 2.2.0. 

So I worked on a POC to check out 2.2.0.M1 and saw that it has a good feature to have a onMessage functionality for streams. Here is a link to POC. https://github.com/akaskj/redispoc

However I noticed that a new thread is created and put to waiting stage so that it can poll the stream for every subscription we make to a stream (Attached screenshots). This will not be scalable with our use case as we may be listening to thousands of streams at a given time. So we cannot have that many treads at a given time. Here is the link to [StreamPollTask|https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/stream/StreamPollTask.java#L129]

My proposal is to have a better threading design to poll streams where we can have limited number(configurable) of threads for polling and we distribute the task of polling among them.

Could you please let me know if this is something I can make changes and contribute back to Spring-data-redis?

Or was there is a use case that you had in mind for the design that is done currently?

Or am I doing anything wrong?

Anyway please let me know and I can send you more details on the design.

 

Thank you,

Akash

 


Affects: 2.2 M2 (Moore)

Reference URL: https://github.com/akaskj/redispoc

Attachments:

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Mar 28, 2019

Mark Paluch commented

Hey Akash,

Our current design resembles Pub/Sub MessageListener design with a few differences. Redis Streams consumption is a bit different in the sense that we do not receive messages but we have to poll for these. We're happy to discuss improvements and we'd love to improve.

Do you have a specific model on your mind? Did you have a look on the reactive Streams API that requires less threads to work with?

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Mar 28, 2019

Akash Kengunte Jayachandra commented

Thank you Mark for your reply.

Reactive Streams API does use significantly lesser threads. I will take a look and see if we can use that instead of non-reactive.

Also it would be very helpful if I could know when is the 2.2.0 expected to release? could you please let me know?

 

 

 

@spring-projects-issues spring-projects-issues added status: declined type: enhancement in: core labels Dec 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: core status: declined type: enhancement
Projects
None yet
Development

No branches or pull requests

2 participants