chore(atomix): refactor cluster event test

[deepthidevaki]

Description

Refactor DefaultClusterEventTest to remove all thread.sleep() and introduce deterministic waiting.

Related issues

closes #4307

Pull Request Checklist

• All commit messages match our commit message guidelines
• The submitting code follows our code style
• If submitting code, please run mvn clean install -DskipTests locally before committing

[Zelldon]

😄 actually I created today a branch where I removed the DefaultEventService 😅 maybe we can discuss if we really want to keep it

[Zelldon]

https://github.com/zeebe-io/zeebe/tree/zell-rm-default-event-service

[deepthidevaki]

Yes. That module has some bugs now. But I like the general idea of it that we can subscribe to specific events and broadcast will only send to that specific receivers instead of sending to the whole cluster.

[npepinpe]

Don't we use it for long polling? But I agree, it's knowingly buggy, so it's arguable if we want to keep it and fix it, or just get rid of it and use something else.

EDIT: could we fix it instead? I remember refactoring it wasn't really hard, just a bit time consuming to write proper tests for the new version.

[Zelldon]

I mean it is a just a buggy wrapper around the clusterCommunicationService. I was able to replace everything so 🤷

[deepthidevaki]

The behavior is different. In communicationService a broadcast send the message to all nodes in the cluster. But the eventService broadcast send the messages only to subscribers.

[npepinpe]

Should we Zoom? 🚀

[Zelldon]

lets do it on monday

[deepthidevaki]

Sending to only subscribers is useful in long polling notifications since only gateways are interested in those messages. Not sure how much overhead it has in a large cluster if every broker is broadcasting this to every other broker and gateways.
For deployment we can get away with out the event service and use the communication service because the sender knows who are the leaders and unicast to them instead of broadcast. This is what we did for snapshot replication.

The current issue in DefaultClusterEventService is that the gossiping subscribers info is not implemented correctly and after a node restart the subscription info is lost and the nodes do not re-gossip them. I discussed with @npepinpe and one way to fix it is to use the membership protocol to propagate the subscribers info. Both SWIM and heartbeat protocol in atomix propagates the member properties and we can add the topics each member subscribes in its properties. Use the gossip provided by the membership protocol instead of using the buggy custom gossip implemented in DefaultClusterEventService.

I think it is a useful abstraction and we should fix it and use it. Let's discuss it on monday.

[Zelldon]

@deepthidevaki this is superseded by #4387 right?

[deepthidevaki]

Replaced by #4387