-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tracking: Message processing pipeline #268
Comments
Try to layout the ideal message pipeline after receiving a gossip. Message Pipelining in SDKHandle general Waku messages received
Message Pipelining in RadiosThe majority of message processing falls on specific radios and their intended application. While the operations are mostly async, there are some improvements worth considering:
Sequence of process sequenceDiagram
participant GP as Graphcast Peer
participant SDK as Graphcast agent (SDK)
participant RO as Radio Operator
participant S as Stateful
GP->>SDK: Waku Signal
SDK->>SDK: Parse signal to WakuMessage
SDK->>RO: Send WakuMessage to Radio operator
Note over RO: Sequential Validation
alt Concurrent
RO->>RO: Different tasks for each message types
else Parallel
RO->>RO: Independently process messages of the same type
end
alt Persistence - In-memory
RO->>S: Update to contain recent typed messages
else Persistence - Retent FIFO
RO->>S: Prune old Messages
RO->>S: Store new Messages
end
S ->> S: Update aggregated summary
RO ->> S: Query summary for periodic report
A different perspective on messages timeline
title Message lifespan
section Generic
Trigger: Either manual or perodic interval based on Networks
Create : Radio Operator collect necessary info
Send : Graphcast agent wraps message into WakuMessage
: Stores a temporary local copy
Waku signal propagation: Send to connected gossip peers
Receive : Graphcast agent receives signal
: Parse raw message into generic Graphcast message
: Graphcast field validation (nonce, timestamp, first time sender)
: Pass message to Radio Operator
Process: Radio Operator handles application specific logic
section Listener Radio
Storage: Asynchronously store generic graphcast message to database
: Summarize messages received/stored
section Subgraph Radio
Further decoding: Try decode messages to PublicPOIMessage
: Validate POI message fields (block hash, sender identity)
: Queue in remote_messages
Message comparison : determine comparison point from local copy
: trigger after message collection duration
: Take local and remote messages with matching deployment and block number
: Determine comparison results for NotFound, Diverged, Matched
: Notification
: Summarize results
|
Nice start! There are some aspects of this work that are missing from the spec, including:
I recognise some of these may be Radio-level concerns, but we should have a spec for message pipelining that factors in these requirements. |
Updated the previous comment to include retention, persistence, aggregations, and pruning. Linking 2 DB issues |
No description provided.
The text was updated successfully, but these errors were encountered: