Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new quorum call option with strict ordering #52

Closed
wants to merge 9 commits into from

Conversation

johningve
Copy link
Member

This pull request adds a new quorum call option, qc_strict_ordering, that has strict ordering of messages due to its use of streams. The code is being used for an implementation of HotStuff in https://github.com/relab/hotstuff (Currently a private repo, not sure who has access).

The code is similar to the stream based approach suggested by @meling in #16. The major difference is that each node runs two goroutines on Node.connect():

  • A goroutine that takes messages from the Node's Send channel and sends them on the stream.
  • A goroutine that receives messages from the stream and puts the messages on a channel corresponding to the ID of the message.

The reason for using these two goroutines per Node is that it avoids waiting for a response. In other words, multiple messages can be sent to a Node independently of how many replies it has sent back.

Because of this behavior it is necessary to give each request/response pair a unique ID. I did not find a clever way to embed this ID field in the messages. Thus it is currently required that the user specifies the name of a common field in both the request and response fields of a method with an unsigned integer type that can be used by gorums. See the example below:

service OrderedStorage {
	// WriteOrdered is a synchronous quorum call that enforces a strict
	// ordering of the messages that are sent to a node, such that
	// a quorum call Q1 will deliver its messages to the nodes
	// respectively before a later quorum call Q2 delivers its messages.
	// The order of replies is not guaranteed for concurrent quorum calls.
	rpc WriteOrdered(stream State) returns (stream WriteResponse) {
		// The option specifies a common field in the `State` and `WriteResponse`
		// messages of type an unsigned integer type that gorums can use to keep
		// track of which requests and replies belong together.
		option (gorums.qc_strict_ordering) = "GorumsMessageID";
	}
}

message State {
	string Value 	= 1;
	int64 Timestamp = 2;
	uint32 GorumsMessageID = 3;
}

message WriteResponse {
	bool New = 1;
	uint32 GorumsMessageID = 2;
}

A counter in Manager is used to generate the message IDs.
Additionally, a map of message IDs to channels is used to provide a way for Nodes to send the response messages back to the quorum call function.

A helper function is generated to simplify the server implementation:

// StorageServerBasic implements the GRPC server API
// WriteOrdered implements the RPC `WriteOrdered`
func (s *StorageServerBasic) WriteOrdered(srv Storage_WriteOrderedServer) error {
	return WriteOrderedServerLoop(srv, func(req *State) *WriteResponse {
		// process state and return a *WriteResponse
	})
}

State is now stored in the manager
The state has to be copied to prevent it from changing before it is
sent to the server.
When implementing the GPRC server API, the generated `ServerLoop`
function can be used to do the sending and receiving of messages.
The ServerLoop simply calls a provided `getResponse` method to get the
response message.
@meling meling self-assigned this Feb 12, 2020
@johningve
Copy link
Member Author

Closing this PR in favor of #55

@johningve johningve closed this Mar 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants