Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to stream filtered values ? #48

Open
kant777 opened this issue Jan 26, 2022 · 1 comment
Open

How to stream filtered values ? #48

kant777 opened this issue Jan 26, 2022 · 1 comment

Comments

@kant777
Copy link

kant777 commented Jan 26, 2022

Imagine I get a stream of messages like below from a incoming socket

{"name": "Alice", "grandslams": 20, "country": "switzerland"}
{"name": "Bob", "grandslams": 10, "country": "spain"}
{"name": "Charlie", "grandslams": 12, "country": "serbia"}

Now I want to filter all the messages that satisfies a certain criteria and push it to an outgoing socket in a streaming fashion. Like say stream all messages where grandslams > 10 and country = "spain". While I am aware few ways to implement this I wonder how to do it effectively using this library?

@kelindar
Copy link
Owner

This library might not be the best tool for the job. Now, if you still want to implement something like this, I would do it in a micro-batch fashion:

  1. When receiving messages, keep inserting them into the database
  2. Periodically (every N milliseconds), perform the query, flush the messages and potentially delete the data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants