Allow for full replication

[samliok]

Replication is currently kickstarted when a node receives a finalization for a future round. It then knows it is missing x amount of sequences so it requests them from other nodes. A node will respond with the requested sequences and also send its latest quorum round to ensure the requesting node has the most recent sequence. This works for finalizations, but is not enough since we aren't guaranteed to receive the most up-to-date notarization(without a finalization) or empty notarization. Furthermore, we need to kickstart the process of replication when we receive future Notarizations and EmptyNotarizations since the network might be stuck and waiting for the lagging node to complete the QC.

To do so, we should extend our replication logic to distinguish between replicating finalized sequences and future rounds with Notarizations/EmptyNotarizations. By keeping track of the latestFinalizedSeq and latestKnownRound, our node also knows if it needs to vote on rounds while replicating(noted here).

With this proposed logic, a lagging node would being replication when it receives a EmptyNotarization, Finalization or Notarization from a future round. In the case where a lagging node reconnects into a halted network, it will send an emptyVote message which will be responded to via

Simplex/epoch.go

Line 709 in 2030b91

func (e *Epoch) maybeSendNotarizationOrFinalization(to NodeID, round uint64) {

. This logic should be updated to also send the latestQuorumRound.

[samliok]

Also we currently don't have a way for a replicating node to replicate a tail end of EmptyNotarizations.
Because we delete previous emptyNotarizations once we enter a new round, a replicating node can't query a normal node for past emptyNotarizations. Therefore if it receives EmptyNotarization with round 10, and normal notarization for round 5 there is no way for it guarantee rounds 6-10 where empty notarizations.

[samliok]

I think we talked about this offline, and decided on replicate the empty notarizations by keeping the empty notarizations in memory until we notarize/finalize a round. If we are worried about running out of memory, we can store them on disk or read them from the wal but we are not going to worry about this for now.