diff --git a/platform-sdk/docs/core/dynamic-address-book/DynamicAddressBook-PlatformDesign.drawio.svg b/platform-sdk/docs/core/dynamic-address-book/DynamicAddressBook-PlatformDesign.drawio.svg new file mode 100644 index 000000000000..442f463428b8 --- /dev/null +++ b/platform-sdk/docs/core/dynamic-address-book/DynamicAddressBook-PlatformDesign.drawio.svg @@ -0,0 +1,4 @@ + + + +
RosterDiffForRound
RosterDiffForRound
Long pendingConsensusRound
Long pendingConsensusRound
RosterDiff currentEffectiveRoster
RosterDiff currentEffectiveRoster
RosterDiff
RosterDiff
Roster newRoster
Roster newRoster
long effectiveRound
long effectiveRound
List<NodeId> addedNodes
List<NodeId> addedNodes
List<NodeId> removedNodes
List<NodeId> removedNodes
List<NodeId> changedKeys
List<NodeId> changedKeys
boolean weightChanged
boolean weightChanged
Application on Handle Transaction Thread
Application on Handle Transaction Thread
handleConsensusRound
handleConsensusRound
EventHorizonManager (EHM) 
EventHorizonManager (EHM) 
void accept(RosterForRound)
void accept(RosterForRound)
void accept(NonAncientEventWindow window)
void accept(NonAncientEventWindow window)
Roster getRosterForRound(long round) 
Roster getRosterForRound(long round) 
Roster getCurrentEffectiveRoster()
Roster getCurrentEffectiveRoster()
void addFutureEventHorizonConsumer(Consumer<Long>  consumer)
void addFutureEventHorizonConsumer(Consumer<Long>  consumer)
void addPendingRoundConsumer(Consumer<Long> consumer) 
void addPendingRoundConsumer(Consumer<Long> consumer) 
void addPastEventHorizonConsumer(Consumer<lLong> consumer)
void addPastEventHorizonConsumer(Consumer<lLong> consumer)
Platform
Platform
Buffered Validation
Buffered Validation
Event Round Buffer
Event Round Bu...
Signature
Validator
Signature...
Orphan
Buffer
Orphan...
EHM
EHM
Parent Time
Validation
Parent Time...
EHM
EHM
Event Intake
Event Intake
preConsensus
preConsensus
preHandle
preHandle
consensus
consensus
Event
Observer
Event...
handle
Consensus
Rounds
handle...
handle
stale
events
handle...
Platform.setNewRoster(Roster) 
Platform.setNewRoster(Roster) 
StateRosterStore
StateRosterStore
Event Validation
Event Validation
Ancient
Validator
Ancient...
Event
Deduplication
Event...
Parent
Validator
Parent...
Transaction
Size Validator
Transaction...
EHM
EHM
Load from state at startup
Load from state...
Event Creation
Event Creation
Tipset
Event
Creation
Manager
Tipset...
Tipset
Event
Creator
Impl
Tipset...
Tipset
Metrics
Tipset...
Tipset
Tracker
Tipset...
Tipset
Weight
Calculator
Tipset...
Gossip
Gossip
GossipFactory
GossipFactory
Chatter
Gossip
Chatter...
Sync
Gossip
Sync...
Abstract
Gossip
Abstract...
Network
Topology
Network...
Connection
Managers
Connection...
Outbound
Manager
Outbound...
Inbound
Manager
Inbound...
Fallen
Behind
Fallen...
Shadow
Graph
Shadow...
Reconnect
Learner
Reconnect...
EHM
EHM
Sync
Manager
Sync...
EHM
EHM
EventTask
Creator
EventTask...
Network
Reconnect
Metrics
Network...
Consensus
Consensus
Consensus
Impl
Consensus...
EHM
EHM
Consensus
Rounds
Consensus...
EHM
EHM
ISS Detection
ISS Detection
Consensus
HashManager
Consensus...
EHM
EHM
NetworkManager Refactor
NetworkManager Refactor

Start of day, all rosters in the saved EventHorizon are broadcast in order of round.
Start of day, all rosters in the saved EventHorizo...
Metrics
Metrics
Network
Network
ISS
ISS
Reconnect
Reconnect
RosterForRound FutureEventHorizon
RosterForRound FutureEventHorizon
NonAncientEventWindow PastEventHorizon
NonAncientEventWindow PastEventHorizon
RosterDiffForRound for pendingConsensusRound
RosterDiffForRound for pendingConsensusRound
RosterForRound
RosterForRound
Long round
Long round
Roster roster
Roster roster
NonAncientEventWindow
NonAncientEventWindow
Long latestConsensusRound
Long latestConsensusRound
Long minRoundNonAncient
Long minRoundNonAncient
Text is not SVG - cannot display
\ No newline at end of file diff --git a/platform-sdk/docs/core/dynamic-address-book/DynamicAddressBook-PlatformDesign.md b/platform-sdk/docs/core/dynamic-address-book/DynamicAddressBook-PlatformDesign.md new file mode 100644 index 000000000000..8ecb99d0386d --- /dev/null +++ b/platform-sdk/docs/core/dynamic-address-book/DynamicAddressBook-PlatformDesign.md @@ -0,0 +1,134 @@ +# Dynamic Address Book - Platform Design + +![Dynamic Address Book Platform Design](DynamicAddressBook-PlatformDesign.drawio.svg) + +## Network Connection + +### Add Node + +Existing nodes need to create or receive mutual TLS gossip connections with the added node. + +This change takes effect when a roster with an added node becomes the `currentEffectiveRoster` for +the `pendingConsensusRound`. + +### Remove Node + +Existing nodes need to stop gossiping with the removed node. Events from the removed node can still be gossiped, +validated and come to consensus until the last roster that includes the removed node becomes ancient. + +This change takes effect when a roster with a removed node becomes the `currentEffectiveRoster` for +the `pendingConsensusRound`. + +### Rotate Signing Key + +All nodes need to tear down their existing mutual TLS connection with the node that has had its key rotated and +establish a new connection with the node using the new signing certificate as the trusted root certificate. + +This change takes effect when a roster with the node's key changes becomes the `currentEffectiveRoster` for +the `pendingConsensusRound`. + +### Node Weight Change + +This has no effect on the existence of network connections. + +## Event Creation + +### Add Node + +Begin using events from the node as parents of new events. + +This change takes effect when a roster with an added node becomes the `currentEffectiveRoster` for +the `pendingConsensusRound`. + +This change will modify network membership and the event creation algorithm will be reset. + +The event creator can build on top of any event that has a birth round that is less than or equal to the event creator's +known value for the `pendingConsensusRound`. Events that have a higher birth round need to be buffered until the event +creator's known `pendingConsensusRound` becomes equal to or higher than their birth round. + +Buffer: Only the latest events from each node within the same birthRound need to be buffered. + +### Remove Node + +When a roster with a removed node becomes the `currentEffectiveRoster` for the `pendingConsensusRound`, the event +creation algorithm is reset. + +### Rotate Signing Key + +The `currentEffectiveRoster` contains the key used to sign the event. Rotation of the roster will update the signing +key to use for signing self events. + +### Node Weight Change + +Changing network weight in the `currentEffectiveRoster` for the `pendingConsensusRound` should cause the event creation +algorithm to reset. + +## Gossip + +### Receiving Events + +The `futureEventHorizon` and the `pastEventHorizon` define the thresholds for receiving gossipped Events. Received +events with `birthRound` outside of this node's Event Horizon will be rejected or discarded. + +### Sending Events + +A node will store ancient events and send them through gossip to help nodes which are behind. + +### Add Node + +When transaction handling creates a roster with a new node in it, it becomes the `greatestEffectiveRoster` for +the `futureEventHorizon` and this node will be able to receive these events. + +### Remove Node + +When a node is removed from a roster and all rosters containing the node have become ancient, this node will no longer +receive events from the removed node. If received, they will be discarded. + +### Rotate Signing Key + +This roster change has no effect on whether or not send or receive events through Gossip. + +### Node Weight Change + +This roster change has no effect on whether or not send or receive events through Gossip. + +## Event Validation + +For each event, lookup the roster within the `eventHorizon` that matches the `birthRound` of the event. Events +outside the `eventHorizon` are not validatable. + +## Orphan Buffer + +Drop ancient events that have a birthRound less than `minRoundNonAncient`. + +## ISS Detection + +Use the effective roster for the round in question. + +## Consensus Effects + +Uses the `currentEffectiveRoster` for the `pendingConsensusRound`. + +When a round comes to consensus: + +* The `latestConsensusRound` and `latestConsensRoster` can be retrieved from the produced `ConsensusRound`. +* A new `NonAncientEventWindow` is produced on an output wire. +* Consensus advances the `pendingConsensusRound` and uses the `currentEffectiveRoster` for that round. +* The `RosterDiff` between the `currentEffectiveRoster` and the `latestConsensusRoster` is produced on an output wire. + +## Signature Gathering for State + +Use the effective roster for the relevant round when validating signatures on the round's state. + +## Reconnect + +Validating state signatures. + +Use the effective roster for the relevant round when validating signatures on the round's state. + +NOTE: The node must be able to validate >1/2 of the weight with its current effective roster. Since >2/3 of the +weight is needed to sign the state, a reconnecting node can tolerate 1/6 of variance in weight assigned to nodes. + +## Metrics + +Most metrics will need to stay in sync with the `currentEffectiveRoster` for the `pendingConsensusRound`. \ No newline at end of file diff --git a/platform-sdk/docs/core/dynamic-address-book/roster/EventWindowTerminology.drawio.svg b/platform-sdk/docs/core/dynamic-address-book/roster/EventWindowTerminology.drawio.svg new file mode 100644 index 000000000000..5abb110ed8d0 --- /dev/null +++ b/platform-sdk/docs/core/dynamic-address-book/roster/EventWindowTerminology.drawio.svg @@ -0,0 +1,4 @@ + + + +
Consensus: Non-Ancient Event Window
Consensus: Non-...

latestConsensusRound
latestConsensusRound
pendingConsensusRound
pendingConsensusRound
Ancient Rounds
Ancient Ro...
latestHandledRound
latestHand...
greatestEffectiveRoster
greatestEffectiveRoster
Gossip: Validatable Event Window
Within The "EventHorizon" 
Gossip: Validat...
Latest Handled Round Creates
Greatest Effective Roster
Latest Handled Round Creates...
Roster Offset
Roster Offset
currentEffectiveRoster
currentEffectiveRoster
maxGossipRound
maxGossipRound
minGossipRound
minGossipRound
minRoundNonAncient
minRoundNonAncient
Rounds
Rounds
Rosters For Rounds
Rosters For Rounds
latestConsensusRoster
latestConsensusRoster
minRosterNonAncient
minRosterNonAncient
futureEventHorizon
futureEventHorizon
pastEventHorizon
pastEventHorizon
Text is not SVG - cannot display
\ No newline at end of file diff --git a/platform-sdk/docs/core/dynamic-address-book/roster/RosterTerminologyDiagram.drawio.svg b/platform-sdk/docs/core/dynamic-address-book/roster/RosterTerminologyDiagram.drawio.svg new file mode 100644 index 000000000000..5d565771f61d --- /dev/null +++ b/platform-sdk/docs/core/dynamic-address-book/roster/RosterTerminologyDiagram.drawio.svg @@ -0,0 +1,4 @@ + + + +
creates
creates
Roster
Effective Round: 100
Roster...
Roster
Effective Round: 99
Roster...
Roster
Effective Round: 98
Roster...
Roster
Effective Round: 97
Roster...
Roster
Effective Round: 75
Roster...
Roster
Effective Round: 74
Roster...
Round 100
Round 100
Round 99
Round 99
Round 98
Round 98
Round 97
Round 97
Round 75
Round 75
Round 74
Round 74
Roster Store
Roster Round Offset: 2
Roster Store...
creates
creates
Roster
Effective Round: 101
Roster...
creates
creates
Non Ancient
Non Ancient
Ancient
Ancient
Roster
Effective Round: 73
Roster...
Round 73
Round 73
creates
creates
The consensus algorithm is trying to reach consensus on round 101 (the pending consensus round) using the roster effective for round 101.
The consensus algorithm is trying to r...
Roster
Effective Round: 102
Roster...
creates
creates
BR 102
BR 102
BR 103
BR 103
BR 95
BR 95
BR 74
BR 74
Event Validation
Event Validation
Events with Birth Round (BR):
Events with Birth Round (BR):
BR 102
BR 102
BR 95
BR 95
Event with BR 103 is rejected because we do not yet have a roster with which to validate its signature and creator membership.

Event with BR 74 is rejected because the birth round is ancient.
Event with BR 103 is rejected b...
Event Creator
Event Creator
BR 101
BR 101
The event creator creates events with a birth round of 101, because that is the roster currently being used to calculate consensus. Any node that has applied the transactions of round 99 will have the roster required to validate this event.
The event creator creates events wit...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/platform-sdk/docs/core/dynamic-address-book/roster/glossary-of-terms.md b/platform-sdk/docs/core/dynamic-address-book/roster/glossary-of-terms.md new file mode 100644 index 000000000000..195b0ca2ad18 --- /dev/null +++ b/platform-sdk/docs/core/dynamic-address-book/roster/glossary-of-terms.md @@ -0,0 +1,155 @@ +# Roster Glossary of Terms + +![Event Window Terminology](./EventWindowTerminology.drawio.svg) +![Roster Terminology Diagram](./RosterTerminologyDiagram.drawio.svg) + +## Round Terminology + +### Latest Consensus Round + +The `latestConsensusRound` is the most recent round to have reached consensus. Equal to the pending consensus round - 1. + +### Pending Consensus Round + +The `pendingConsensusRound` is the round that the consensus algorithm is currently attempting to reach consensus on. +Equal to the latest consensus round + 1. + +### Rounds Non-Ancient + +A setting. `roundsNonAncient` is the number of rounds that have reached consensus that are not ancient. + +### Ancient + +An event is ancient if its `birthRound` is less than or equal to the `latestConsensusRound` - `roundsNonAncient`. + +### Minimum Round Non-Ancient + +`minRoundNonAncient` is the minimum round that is considered non-ancient. This is equal to +`pendingConsensusRound` - `roundsNonAncient`. + +### Non-Ancient Event Window + +Specifies the window of events that are being considered in consensus. The window specifies the following: + +* `latestConsensusRound` - the latest round that has reached consensus. +* `minRoundNonAncient` - the minimum round that is considered non-ancient. + +### Latest Handled Round + +The `latestHandledRound` is the latest round to have been completely handled and applied to the state by the node. +Note that the application of this round to state produces a new roster to become effective after a configured +`rosterOffset`. see terminology below. + +### Future Event Horizon (a.k.a. Max Gossip Round) + +The `futureEventHorizon` is the maximum round for which the node has an effective roster. It is also the +maximum round that this node can gossip about. Events which have a birth round higher than the `futureEventHorizon` +will not be gossipped. + +### Past Event Horizon (a.k.a. Min Gossip Round) + +The `pastEventHorizon`, equal to `minRoundNonAncient`, is the minimum round for which the node has an effective roster. +It is also the minimum round that this node can receive gossip for. Events which have a birth round lower than the +`pastEventHorizon` are ancient. Ancient events are retained in gossip for a period of time to support peers that are +behind in consensus. + +### Validatable Event Window (within the Event Horizon) + +The `validatableEventWindow` is the window of rounds which have effective rosters that have not become ancient. It +spans, inclusively, from the `pastEventHorizon` to the `futureEventHorizon`. Events with `birthRound` within this +window are non-ancient and can be validated. + +## Roster Terminology + +### Roster (a.k.a. Consensus Roster) + +The collection of nodes which create events and participate in consensus. Prior to the dynamic address book effort, +this was called the `AddressBook` in the platform's code. The name was changed to `Roster` to disambiguate it from +the application's concept of an address book. The application's address book contains the consensus roster. + +### Effective Roster (for a round) + +The roster that was used to determine consensus for a particular round. The `effective roster for round X` is the +roster that was used to compute consensus for round X. + +The effective roster is used for more than just consensus: + +* **Networking** - determine who to connect to and facilitate mutual TLS connection with that node. +* **Event Creation** - determine which other nodes' events are available to use as parents of events. +* **Event Validation** - validate the signature using the signing key of the node that created the event. +* **State Validation** - validating the signatures on the state of a round using the signatures in the effective + roster for that round. + +### Roster's Effective Round + +The round that the roster will be used to calculate consensus for. This is the round number used to look up the +appropriate roster to validate an event. The event contains this round number as the event's `birthRound` and is +used to uniquely identify the roster that should be used to validate the event. + +Not a field inside the roster. + +### Roster's Creation Round + +The round of transactions that created this roster once they were applied to the state. + +### Roster Offset + +A setting. The number of rounds it takes for a roster to become effective after being created. This is likely to be +a small integer value, e.g. 2. If this number is 2, then processing all the transactions in round 100 yields the +effective roster for round 102. + +### Ancient Roster + +All rosters that are effective for ancient rounds are ancient rosters. + +### Minimum Roster Non-Ancient + +`minRosterNonAncient` is the roster that is effective for `minRoundNonAncient`. + +### Latest Handled Roster + +`latestHandledRoster` is the roster that was effective and used to calculate consensus for the `latestHandledRound`. + +### Latest Consensus Roster + +`latestConsensusRoster` is the roster that is effective for the `latestConsensusRound`. + +### Current Effective Roster + +`currentEffectiveRoster` is the roster that is effective for the `pendingConsensusRound`. It is currently being +used by consensus and determines the current network topology. + +### Greatest Effective Roster + +`greatestEffectiveRoster` is the roster that is effective for the `futureEventHorizon`. It is the greatest roster +to have been produced by handling transactions and applying them to the state. + +## Event Terminology + +### Event Fields + +Events have four parts to them: + +1. Hashed Data +2. Signature +3. Consensus scratchpad +4. Generated by consensus + +### Birth Round + +`birthRound` - The pending consensus round as known by the event creator thread at the time the event was created. + +This field allows nodes to determine if the event is ancient or not. + +Uniquely defines the roster that should be used when validating this event. An event with birth round X should be +validated using the roster with an effective round of X. + +### Consensus Round + +`consensusRound` - The round in which an event reaches consensus. Previously known as `roundReceived`. + +### Current Round + +Used only in the consensus algorithm. Previously known as `roundCreated`. In the future, this data will +not be exposed outside of consensus. +