Proposal: gRPC Refactoring

[asher-pem-arm]

To facilitate future extensions to Labgrid with functionality such as authentication/authorization as well as to improve the experience integrating Labgrid with external systems, we propose as suite of backwards compatible extensions and refactoring to the gRPC API.

The proposal includes the following groups of changes:

1. Deprecation of the StartupDone message (used in the ClientStream and ExporterStream) and metadata-based identity passing
   The current API relies on the ClientStream being created and the StartupDone message being sent before the client is able to successfully issue some other RPCs. The requirement to setup and maintain the ClientStream adds complexity to 3rd party clients and integrations that wish to interact with the Labgrid API.
   We propose the deprecation of the StartupDone message and a replacement mechanism, gRPC metadata, for transmitting the client identity and version. The changes will ensure backwards compatibility for older clients that still send their identity over the ClientStream.
   Corresponding changes are proposed for the ExporterStream.
2. Creation of new unary RPCs
   Certain functionality, like listing the Resources on the Coordinator is only available by processing the updates provided over the ClientStream. We propose adding explicit unary RPCs to achieve this functionality. This should simplify the labgrid-client codebase and allow for easier integration with 3rd party clients.
3. Standardisation of the existing unary RPCs
   We propose the standardisation of the existing unary RPCs to improve naming and functionality (e.g. pagination). We have based our proposals, partly, on Google's AIPs.

Deprecation of the StartupDone message (used in the ClientStream and ExporterStream)

Overview

The successful transmission of the StartupDone message over the ClientStream is a prerequisite for a number of other RPCs to succeed (AddPlace, AcquirePlace, AllowPlace, CreateReservation).

Proposal

We propose replacing the StartupDone message with three gRPC metadata KV pairs that are transmitted on each RPC call:

Key	Description of Value
x-lg-username	The username of the user calling labgrid-client. This KV pair is not relevant to labgrid-exporter.
x-lg-hostname	The hostname of the machine which labgrid-client or labgrid-exporter is being used from.
x-lg-user-agent	The type and version of the client (eg. labgrid-client or a 3rd party client or integration) that is making the RPC call. This key MAY be omitted when the API is called by an entity other than labgrid-client.

Implementation

The labgrid-coordinator implementation MUST retain the ability to infer user identity from the StartupDone message to be backwards compatible with older labgrid-client versions.

The labgrid-client and labgrid-exporter no longer send the StartupDone message and instead uses the metadata KV pairs described above.

This change will remove the need for a system that directly consumes the gRPC API to establish a ClientStream before calling the unary RPCs. This reduces the amount of state-management logic needed on clients, making integration with Labgrid easier.

Creation of new unary RPCs

Overview

We propose transitioning the API to use unary RPCs where possible, in a backwards compatible way. Unary RPCs have the benefit that they are easier to use within client code compared to stream RPCs due to less state-handling being needed. The new unary RPCs are more tightly scoped to make the future implementation of authentication/authorization functionality easier. This change is facilitated by the deprecation of the StartupDone message.

Proposal

The following new RPCs are proposed:

RPC	Purpose	Rationale
GetPlace	Get the details of a single Place.	You can currently only list all Places. This could be costly for larger deployments of Labgrid with many places per labgrid-coordinator. It is simpler for a client interested in a single Place to query the information about that Place directly, rather than listing all Places and having to filter for the desired Place on the client side.
ListResources	List the available Resources.	You can currently only list Resources by establishing a ClientStream and populating a local cache based on the labgrid-coordinator-sent ClientOutMessages. This RPC simplifies this process for clients that do not need real-time updates and would like to easily list all Resources on the labgrid-coordinator.
ListPlaceResources	List the Resources associated with a Place.	You can currently only see the Resources associated with a place by establishing a ClientStream and subscribing for all resource updates. This RPC simplifies this process for clients that do not need real-time updates and would like to easily list all Resources for an acquired place on the labgrid-coordinator.
UnsharePlace	Remove a user's access to a Place that they previously had access to via a share/allow.	There is currently no counterpart to AllowPlace (which will be superseded by SharePlace, described below). This RPC adds the ability to remove a user that was previously added with AllowPlace/SharePlace.

Standardisation of the existing unary RPCs

Overview

We propose a combination of replacing RPCs and the addition of renamed fields in existing messages to create a more uniform API. For backwards compatibility purposes, the existing RPCs will be deprecated but not removed.

The naming of RPCs, message fields and the implementation of functionality such as pagination is influenced by Google's AIPs.

Proposal

Creation of new RPCs to replace existing RPCs

Existing RPC to deprecate	New RPC	Proposed definition	Functional differences to existing RPC
AddPlace	CreatePlace	service Coordinator { rpc CreatePlace(CreatePlaceRequest) returns (CreatePlaceResponse) {} } message CreatePlaceRequest { string name = 1; } message CreatePlaceResponse { Place place = 1; }	Returns the newly created Place in the RPC response
GetPlaces	ListPlaces	service Coordinator { rpc ListPlaces(ListPlacesRequest) returns (ListPlacesResponse) {} } message ListPlacesRequest { optional string filter = 1; int32 page_size = 2; string page_token = 3; } message ListPlacesResponse { repeated Place places = 1; string next_page_token = 2; } https://google.aip.dev/158	Supports filtering Supports pagination
AllowPlace	SharePlace	service Coordinator { rpc SharePlace(SharePlaceRequest) returns (SharePlaceResponse) {} } message SharePlaceRequest { string name = 1; string user = 2; } message SharePlaceResponse {}	None
PollReservation	RefreshReservation	service Coordinator { rpc RefreshReservation(RefreshReservationRequest) returns (RefreshReservationResponse) {} } message RefreshReservationRequest { string reservation_id = 1; } message RefreshReservationResponse { Reservation reservation = 1; }	None
GetReservations	ListReservations	service Coordinator { rpc ListReservations(ListReservationsRequest) returns (ListReservationsResponse) {} } message ListReservationsRequest { optional string filter = 1; int32 page_size = 2; string page_token = 3; } message ListReservationsResponse { repeated Reservation reservations = 1; string next_page_token = 2; }	Supports filtering Supports pagination

Clarifications of RPC message field names and types

We believe some of the existing field names have ambiguous names which could cause confusion. Additionally, the handling of timestamps can be improved.

As renaming of fields in Protobuf messages causes changes to the JSON serialisation and generated stubs, we cannot make changes to the field names without potentially affecting other direct users of the gRPC API. Instead, we suggest adding new fields with improved names and deprecating the old fields. The implementation will need to continue to support the old fields to maintain backwards compatibility with older clients.

Protobuf Message	Existing Schema	New Schema	Rationale
Reservation	message Reservation { ... string token = 2; ... double created = 7; double timeout = 8; ... }	message Reservation { ... string token = 2 [deprecated = true]; ... double created = 7 [deprecated = true]; double timeout = 8 [deprecated = true]; ... string id = 9; google.protobuf.Timestamp create_time = 10; google.protobuf.Timestamp expire_time = 11; }	token can be misunderstood to mean something that relates to authentication/authorization functionality. The Reservation token acts as an identifier for the Reservation, so we propose making this unambiguous by changing it to id.
Place	message Place { ... optional string acquired = 6; ... repeated string allowed = 8; double created = 9; double changed = 10; ... }	message Place { ... optional string acquired = 6 [deprecated = true]; ... repeated string allowed = 8 [deprecated = true]; double created = 9 [deprecated = true]; double changed = 10 [deprecated = true]; ... optional string owner = 12; repeated string shared_with = 13; google.protobuf.Timestamp create_time = 14; google.protobuf.Timestamp update_time = 15; }	We believe these field names improve the readability and clarity of the functionality being provided.