[Feat.Req] Message pipelining

[hopeyen]

Listener Radio

Problem Statement

To efficiently capture, validate, and record various types of radio payload messages, including public POI messages, the Listener Radio plays a pivotal role in understanding a namespace in the Graphcast Network. Its main responsibility is to monitor the integrity and robustness of the network by effectively listening to broadcasts and storing valid messages in a dedicated database.

Capturing Broadcast Messages

Listener Radio remains constantly active and receptive to all broadcasted messages within the Network. Irrespective of the message type, Listener Radio intercepts each message, ensuring no message goes unnoticed.

• TODO: Listener radio currently stores a set of supported message types. It should eventually store all messages with compatible traits for GraphcastMessages.

Validating the Messages

On receiving a message, the Listener Radio invokes a series of validation checks:

1. Authentication: Ensures that the message is from a trusted and legitimate source within the network.
2. Integrity: Confirms that the message hasn't been altered or tampered with during transmission.
3. Duplication: Checks the database to ensure the message hasn't been recorded previously, preventing double entries.
4. Nonce check: Ensures a linear ordering of messages.

Storing Validated Messages

Once a message passes all validation checks, Listener Radio processes it for storage. The storage database is structured to store different types of messages in a general JSONB column, ensuring quick retrievals and queries in the future.

• Implement an optimization routine to periodically prune older messages, freeing up storage and maintaining database performance.

Message services

The Listener Radio is equipped to handle various payload message types and providing a quick glance of the overall network

1. Network Metrics: Periodic summary about network health, peer counts, active topics, and other vital statistics.
2. HTTP server: Provide API to query overall or specific message entries

Expectation Proposal

Current Database Specifications

1. Message ID: A unique identifier for each captured message.
2. Message: A json object as a exact copy of messages from the network.

• Add potential new fields
  1. Source: Details about the sender or broadcaster of the message.
  2. Type: Classifies the message (e.g., Public POI, Upgrade intent, etc.)
  3. Validation Timestamp: The exact time when the message was validated by the Listener Radio.

Automated Reporting

Generate daily/weekly summaries of captured and stored messages, aiding in monitoring and potential troubleshooting.

• Add a set of metrics

1. Total Messages Captured
   Provides an overall view of the network activity in terms of the number of messages broadcasted and captured by the Listener Radio.
2. Breakdown of Message Types
   Display the number of messages categorized by type to offers a deeper understanding of the nature of network activity.
3. Validation Success Rate
   Percentage of messages that successfully passed the validation checks against those that failed, gauge the integrity of broadcasts in the network.
4. Database Storage Utilization
   Display the current storage usage in terms of total messages stored and available space, such that users can adjust their pruning preferences.
5. Top Active Broadcasters
   List the top entities or nodes that are the most active in broadcasting messages. Helps identifying primary sources of information or potential spamming entities.
6. Average Message Processing Time
   The mean time taken to capture, validate, and store each message as an indicator of the efficiency and responsiveness of the Listener Radio.
7. Recent Message Log
   Display the last few messages with their type, source, and timestamp can provide a quick snapshot of recent network activity.
8. Daily/Weekly Message Volume Trends
   Graphical representation of the number of messages captured over time (daily, weekly).
9. Failed Message Analysis
   To provides reasons or categories for messages that failed validation (e.g., authentication failure, duplication, unsupported messages) for troubleshooting or identifying malicious activities.
10. Network Latency Metrics
    Average time taken for a message to travel from the broadcaster to the Listener Radio can helps users to understand network performance and potential delays.
11. Messages Awaiting Storage
    Count of messages that have been validated but are pending to be stored in the database will indicates potential bottlenecks in storage or processing.
12. Unique Broadcasters Count
    Total number of unique entities or nodes broadcasting messages.
    Gives a sense of the diversity and size of the active network participants.
13. Database Pruning Events
    Track when and how many messages were pruned from the database.

Alternative Considerations

Feedback Mechanism: Allow Listener Radio to send acknowledgments back to the network, sharing an overall status to the peer nodes.