Expose gateway topology

[npepinpe]

Description

It's particularly useful to know what the gateway thinks the topology of the cluster is at a given point in time when debugging, especially when doing post-mortem investigations.

The gateway topology should be exposed as:

1. Metrics: so we can consume them as a time series and find point-in-time state
2. MBean/Actuator: so we can debug a running system (Optional)

[pihme]

For exposing topology through MBean I would recommend using:

• InfoContributor https://www.baeldung.com/spring-boot-info-actuator-custom
• Using SpringGatewayBridge to get a hold of the topology

(Note that the info endpoint currently is deactivated)

Happy to help out if you have questions about the actuators.

Not sure how to expose topology as a metric.

[korthout]

@pihme thanks 👍

[aivinog1]

Hi! I would like to start working on this. I can start implementing an MBean part, after that we can discuss metrics :)

[npepinpe]

Hey @aivinog1, we haven't really done MBean right now. I think @pihme looked into it, and it wasn't all that great? Correct me here if I'm wrong, I only vaguely remember.

My first suggestion would be to implement the metrics in the Topology manager, then add a simple actuator which just returns the topology as JSON, but I'm open to suggestions.

[aivinog1]

@npepinpe Hi! I have a question about how metrics should look like? Should it be the same as the topology response command? For example, the gateway knows about 2 brokers (the first metric), the first broker knows about 2 partitions(so, there must be a second metric), they are both healthy and it is a leader for partition 2 (third metric).

[deepthidevaki]

@npepinpe I don't understand why we need to expose the topology via actuators. The topology is already exposed via zbctl.

For the metrics:
IMO, the interesting metrics to expose is what gateway knows as the leader for each partition. I don't think the whole topology needs to be exposed via metrics. So my proposal is to have one metric that indicates who is the leader for a partition.

[Zelldon]

I agree with @deepthidevaki I would do it similar to the metrics we have in raft

[npepinpe]

Since we already expose the topology as part of our client, you're right, no point making it an actuator.

As for the metrics, definitely the leader is usually the important part, but it can still be useful to know the followers to ensure that we have a consistent view of the cluster. Are we sure that knowing the wrong followers could never lead to an issue or help us diagnose one?

[deepthidevaki]

Gateway doesn't care about followers. We have raft metrics that shows which nodes are followers for each partition. That is enough for debugging, IMO.

[npepinpe]

Couldn't it be useful to know the complete topology from the gateway point of view at times? As in, what if I have no leader? Is the gateway unaware of one, or is it a follower according to it, or...? Couldn't it also allow us to track bugs related to how the topology is built over time by checking the transitions from follower to leader and back according to the gateway?

[deepthidevaki]

Ya. Makes sense.
Then the metrics would show the following table:
GatewayId | Broker | Partition | Role (Leader or Follower)