feat: Support Redis Importer #656
Conversation
|
Hi @VonDerBeck, this sounds great! We are using the Hazelcast integration. Because of the RingBuffer, this may lead to loosing events, which AFAICS is hard to detect and near to impossible to solve afterwards. What do you say, would this issue go away with switching to Redis? And do you know if it works well with high volumes from Zeebe, e.g. the monitor is still able to process incoming events? Thanks in advance! |
|
great to hear from you.
The Redis Exporter uses streams and consumer groups. So there is no risk of loosing events. The exporter is optionally able to regularly clean up acknowledged events. This is way more stable than a RingBuffer and the reason why I prefer Redis. You can even setup Redis with persistence.
Just test your scenario and let me know the results :-). There are some tuning options when it comes to performance. Generally I would not expect it to be slower than Hazelcast. |
|
looks like you have been in charge releasing the latest version of Zeebe Simple Monitor. On the other hand this repository is marked as "in search of a maintainer". Would you like to review my changes? Or should I simply merge it? Typically, when working on such a repo you have a strategy in mind where you like to see this thing in the future and PRs should fit into this long term vision. So your review would be very valuable for me. Just let me know 😃 Thanks in advance |
Hi Gunnar, Regards |
|
Thank you very much for this contribution @VonDerBeck 🚀 |
In order to make Zeebe Simple Monitor even more flexibel, I added the option to import events using Redis.
See https://github.com/camunda-community-hub/zeebe-redis-exporter.
The importer side is very similar to the Hazelcast Importer. So there is a lot reuse of the existing code. I moved the Protobuf specific importer classes to a package where they can be used by both, Hazelcast and Redis. The rest of the changes is simple setup and connection stuff.
Why Redis? Redis (https://redis.io/) offers a lot of options to us. We can use it as fast in-memory technology and simple transport layer. We can optionally add persistence, but it is not required. We have Redis streams at our hand which are great when we want to use multiple consumers and scaling on the event receiver side. If neither the Hazelcast nor the Kafka exporter fits the exact requirements, this could be the one.
If there are any questions and hints, let me know.