Would the metric just be a single counter for each branched event detected or a counter per node? In general I think we need to move away from per-node metrics. Perhaps a single counter is sufficient, paired with a log message that says which creator branched.

Would the metric just be a single counter for each branched event detected or a counter per node? In general I think we need to move away from per-node metrics. Perhaps a single counter is sufficient, paired with a log message that says which creator branched.

We could do either. If a per-node metric is not desirable, we could just go with a single metric for all nodes plus a rate limited log.

What would the metric look like? number of nodes branching? if so, when would this value be reset?

@lpetrovic05

What would the metric look like? number of nodes branching? if so, when would this value be reset?

If it were just a single metric, then I'd just count the number of branches per second. For figuring out which nodes are branching I'd use logging. One rate limited logger per node configured to log at most once per 10 minutes. The metric would alert us to the branching, and we could use the logs to figure out how severe the branching was.

the rate limited logging might not be great. we would detect the first branch, log it, then silence for 10 min. we would not know how many nodes are branching

the rate limited logging might not be great. we would detect the first branch, log it, then silence for 10 min. we would not know how many nodes are branching

Maybe we need to get more details about how we count a branch. Should we would increment a counter every time a new branch is created, and decrement it every time the tip of a branch goes ancient? This would tell us how many times a node created a branch new branch instead of continuing a branched line and would be a single metric.

Would it be worth tracking the number of events in a branch in addition to this?

the rate limited logging might not be great. we would detect the first branch, log it, then silence for 10 min. we would not know how many nodes are branching

I agree, could hide data. But it's probably the best we can do if we don't want to have one metric per node.

Maybe we need to get more details about how we count a branch. Should we would increment a counter every time a new branch is created, and decrement it every time the tip of a branch goes ancient?

This significantly complicates the implementation. My suggestion is to do something simple so that we catch branching in the short term, and consider saving deeper anaysis for a future effort.

Does this box need to get information from hashgraph as to which round is ancient?

Modulo any details around how metrics/logs are counted and reported, the idea is a +1 from me. Will be handy now and the basis of what we need to do in the future.

I suggest a metric like maxNonAncientWeight. Keeps track of the maximum number of weight (as a fraction of total weight) at any given time, this can immediately tell us if an ISS was caused by branching

Does this box need to get information from hashgraph as to which round is ancient?

Yes, we will wire it up to the event window wire. There is a bunch of stuff in the intake pipeline which currently subscribes to that data flow.

I suggest a metric like maxNonAncientWeight. Keeps track of the maximum number of weight (as a fraction of total weight) at any given time, this can immediately tell us if an ISS was caused by branching

I will add this metric.