Add metrics to SwimMembershipProtocol

[deepthidevaki]

Description

To help tune the config parameters of Swim protocol, it would be useful to add some metrics.
Examples:

• Probe latency - RTT for a probe request
• Gossip latency - how long until a metadata change on a node is propagated to another node.

Related to #4827 (comment) #4827 (comment)

[Zelldon]

I feel with adding messaging service metrics we this covered as well #11353 we have labels for the message types, which allows us to see also probe and sync req-response latencies etc.

wdyt @npepinpe

[Zelldon]

I could add a separate section to the dashboard for swim using these metrics

[rodrigo-lourenco-lopes]

Since we are broadcasting the gossip of these updates with no answers, how could we go about measuring this gossip latency?
How about implementing a response for these broadcasts along with the metric and putting everything behind a feature flag?
Or ideally, there is a more straightforward way to measure this.

https://github.com/camunda/zeebe/blob/5280440c43122cf2da5b4b83dbff76a644273e7c/atomix/cluster/src/main/java/io/atomix/cluster/protocol/SwimMembershipProtocol.java#LL753C1-L758C4

wdyt? @npepinpe @Zelldon

[npepinpe]

Honestly, it sounds to me like distributed tracing would be the tool we want to have here. Start an operation somewhere, measure when it completes somewhere else, possibly with hops.

As we don't have that yet, I would postpone this. However, there may be other metrics we'd like just from the local node?

[rodrigo-lourenco-lopes]

The other thing we could measure perhaps is sync() we have a response for this one.

[lenaschoenburg]

We could also try to just export the current state as metrics. Then we could derive additional properties such as "how long does it take to propagate changes throughout the entire cluster" by calculating it on the metrics. Not ideal but I feel like just exporting the current state of SWIM as metrics is useful already.

[rodrigo-lourenco-lopes]

@oleschoenburg If I understood correctly we would export the local state of all members on each node?