[Feat.Req] Message pipelining with persisted operations summary

[hopeyen]

Problem statement

Message pipelining is a crucial point for the success and efficiency of Subgraph Radio's POI cross-checking feature. It ensures that public POIs are seamlessly created, sent, received, processed, and cross-checked. This issue goes over the current state and missing pieces of the pipeline.

Triggering Creation of POI Messages

Upon message block number calculated by a fixed block interval and the synced indexing status, Subgraph Radio operator initiates the process to create a public POI message.

Sending the POI Message

Once a public POI message is created, operator immediately passes the message to Graphcast agent for the message signing and broadcast to the Graphcast Network, ensuring other Indexers are informed about the latest POIs.

Receiving and Processing

Other Indexers' Radios constantly listen for incoming public POI messages. Once a message has been send to the network, they start a local timer for message collection window and accept messages for that message block. When a message is received, it's authenticated and cached for further processing.

• TODO: The message arrival time varies depending on the network connectivity, but make sure that we can accept remote messages sent before our local message if the time period is reasonable.

Cross-checking Calculation

After the collection window for a certain block, the radio compares the local POIs against the received public ones. The goal is to establish consensus across all Indexers (radios are limited to compute consensus from the received messages). If there are any new divergence, alerts are triggered.

Persisted State of Summary for POI Messages

Provide an overview and insight into the behavior and performance of the Subgraph Radio regarding the handling of POI messages, a persisted state of summary is maintained. This summary should enable users to quickly gauge the health, performance, and trends related to the POI messages without diving deep into individual message details.

• We currently have summary logged for sending, comparing, and network updates, but there's a few improvements for better quality and comprehensiveness

Expectation proposal

Summary Specifications (can be extended)

1. Total topics tracked
   • List of IPFS hash for the deployment
2. Total Message Count
   • [ ]A running total of all the POI messages that have been processed. The count is incrementally update by adding the number of new messages. Potentially labeled with deployment hash in metrics and read from metrics.
3. Average message frequency
   • The mean of receiving messages from the network, labeled with deployment hash. This gives us an insight to the message traffic. Update for each message block interval or use a simple moving average $SMA=\frac{x}{N}$ where $x$ is the sum of message counts over the last N periods.
4. Average Processing Time
   • The mean time taken to process each POI message, giving an insight into the efficiency and speed of the Radio. Use equation Σ(Processing times of new messages) / Number of new messages for each block interval. In combination with Average message frequency, we can better understand the performance of radio in relation to the network.
   • Explore ways to utilize autometrics for the functions
5. Total Divergence Detected
   • A count of all instances where a divergence was identified during the cross-checking calculation. A consecutive diverged range of a deployment should be counted only once as metric is incrementally update by adding any new diverged state. Optionally labeled with deployment hash.
6. Frequent senders
   • List of Indexers sorted descending by the number of POI messages received, useful to identify active participants or potential spamming nodes. Update using a mode calculation based on the source of the latest set of messages within a comparison interval.
7. Divergence Rate
   • The percentage of messages that resulted in a discrepancy. Calculated as (Total Divergence Detected / Total Message Count) * 100. Update by recalculating the rate using the updated total divergence and total message count for a given timeframe.
8. Latest Message Timestamp
   • Timestamp of the last received POI message to indicate the most recent activity. Capture the timestamp from the latest message. Can be labeled with deployment hash.
9. POI stakes for Deployment
   • A reference to the stake for the POI that has the maximum on-chain stake backing it for a deployment.
10. Network connectivity

• Aside from the active senders, record the average number of gossip peers to reflect connection with the network

Alternative considerations

More metrics