PDP 21 (Protocol Revisioning)

Protocol revisioning

Status: Implemented

Related issues:

Motivation

Protocol revisioning consists of attributing a version to the collection of messages that form the wire protocol we use for the communication between components in the system. Over time, as we make changes to the protocol, we change the version accordingly so that the different components can understand each other when exchanging messages. We typically bump up the version when making changes to the protocol.

Version compatibility

One of the important aspects of protocol versioning is compatibility. Compatibility is the set of rules that specify what components can talk to each other under what versions. There are at least four possibilities for compatibility:

1. Server is backward compatible. In this model, the server is able to talk to any client that has smaller version. This model is used in a lot off systems because it maps well to the common practice of upgrading the server first and the clients next. One variant of this approach is to use a major and a minor version such that a server exchange messages with any client that has the same major version and the same or a smaller minor version.
2. Client and server have the same version. In this model, the communication between client and server is possible only if they execute the very same version of the protocol. This model is ok, but not desirable because during an upgrade, clients and servers will not be able to talk to each other, and this interruption is often inconvenient or unacceptable.
3. Version ranges: A variant of the first option is that client and server exchange version numbers and agree upon a version to use. A typical way of doing it is to have the client initially sending the versions it supports with a current and oldest compatible, and the server acknowledges with the corresponding version that it supports.

Our recommendation is to implement either 1 or 3.

Changes to the protocol

To be able to add new features to the server in the future we will need to make changes to the protocol for communicating with it. There are different types of changes with different levels of impact.

1. Supplemental information - Additional information is provided on an existing request or reply. This may or may not be used by the recipient, and does not affect the correctness of the application. For instance ids used to trace requests, or metrics could fall into this catagory.
2. New API - A new API on the server side requires new request and reply messages. A server that supports the new API is fully compatible with a client that does not, because the client will never make the call. Similarly a new client could interact with an old server, but would be unable to take advantage of the new API.
3. API change with backward compatible logic - An API on the server is modified, but the old code path is maintained so that older clients can continue to expect the same behavior. (This is very similar to adding an new API but differs in that much of the code for the new API comes from a previous API)
4. New reply type - We do not want to force the client to update first, so the we need to assume that the server might need to send a response while the client does not support it. The server should not send such a message in the case the client does not support it.
5. Totally incompatible change - A change that is a complete revision, the older server cannot work with the new client and vice versa.

We need a mechanism for handling each of these types of changes.

Proposal

To track compatibility there is a version number maintained in WireCommands.java. Each time we make a change to the protocol messages, we increment this number. Incrementing the version number must be present in the same pull request that proposes the protocol change.

When the client and server first connect, they each send the hello command, which contains a range of protocol versions that they support.

To support supplemental information, we can add fields to the end of existing wire protocol types. These will be implicitly ignored if either side is running older code. Even though these changes are fully compatible we should increment the protocol version, and the maximum version supported. Incrementing these versions enable the change to be made non-optional in the future, if for whatever reason that is desired. For example, say that we add a new field to a message and increment the version to 10. At version 10, the field is optional. To make the version not optional, we can simply assign 10 to the lowest compatible version.

To support adding a new API or an API change that is backwards compatible, we can use the protocol revision in the hello message to determine support. Upon connecting, the client sends the range of versions it supports and receives the server range. If we increment the version number of the API every time a new API is added, then when support is added on the server side for the new message type it increments the upper end of its supported range. The client can use this version number to detect if it can invoke the new API. An older client will still work because the server still understands the version it is using, and this is conveyed to it by the range of versions the server supplied in the Hello.

To support a new reply type we can do the reverse, where the upper supported version is incremented by the client when support for the new version is added, and the server can use this to detect if it can send the new reply type.

For a completely incompatible change, the version number for the protocol is incremented and both the upper and lower supported versions on the client and the server are set to this new value. This will cause any client or server using the old version to immediately detect that it is incompatible.

Open questions

• When should we break compatibility?

In some cases we may want to simplify the code after some period of time, so that each side can just assume the information will be present. We could do this, breaking compatibility by incrementing the minimum supported version. This comes with an upgrade complexity for users.

Discarded approaches