Dependencies

Whenever the replicated state machine agrees on a proposal, it passes it to the application layer by invoking Deliver.

Application {
      Deliver(Proposal, []Signature)
}

Each node communicates with the other nodes. To receive messages from Comm nodes implement HandleRequest(sender uint64, req []byte) and HandleMessage(sender uint64, m Message) methods.

Comm {
    SendConsensus(targetID uint64, m Message) 
    SendTransaction(targetID uint64, request []byte) 
    Nodes() []uint64
}

Consensus proposals are different for each application, however, they are all batches of clients' requests and they may contain metadata. The Assembler builds an application specific Proposal, out of the given metadata and requests, that the leader will later propose for consensus. It returns a proposal that consists of some of the requests, and a remainder of the requests that could be proposed again in a new proposal.

Assembler {
    AssembleProposal(metadata []byte, requests [][]byte) (nextProp Proposal, remainder [][]byte)
}

Each node logs its state changes to the disk, in order to restore it after it crashes. The changes are appended consecutively to a file, and upon startup/recovery the file is entirely loaded into memory to restore the internal state. Upon Append we signal the WAL if it is safe to truncate. This library also contains our own WAL implementation.

WriteAheadLog {
    Append(entry []byte, truncateTo bool) error
}

To support different types of signatures, this library is dependent on a Signer.

Signer {
    Sign([]byte) []byte
    SignProposal(Proposal) Signature
}

A Verifier is used to validate the proposals suggested by a leader and to verify signatures.

Verifier {
    VerifyProposal(proposal Proposal) ([]RequestInfo, error)
    VerifyRequest(val []byte) (RequestInfo, error)
    VerifyConsenterSig(signature Signature, prop Proposal) error
    VerifySignature(signature Signature) error
    VerificationSequence() uint64
}

RequestInspector inspects requests and derives properties from them, such as a unique request ID, or a unique ID of the client that issued the request.

RequestInspector {
      RequestID(req []byte) RequestInfo
}

RequestInfo {
     ID string
     ClientID string
}

A Synchronizer reaches the cluster nodes and fetches blocks in order to sync the replica's state. An invocation of Sync() blocks indefinitely until the replica's state is synchronized to the latest decision, and returns it.

Synchronizer {
     Sync() Decision
}