Multi-endpoint feature

[nimf]

The purpose of GcpMultiEndpointChannel is twofold:

1. Fallback to an alternative endpoint (host:port) of a gRPC service when the original endpoint is completely unavailable.
2. Be able to route an RPC call to a specific group of endpoints.

A group of endpoints is called a MultiEndpoint and is essentially a list of endpoints where priority is defined by the position in the list with the first endpoint having top priority. A MultiEndpoint tracks endpoints' availability. When a MultiEndpoint is picked for an RPC call, it picks the top priority endpoint that is currently available. More information on the MultiEndpoint class.

GcpMultiEndpointChannel can have one or more MultiEndpoint identified by its name -- arbitrary string provided in the GcpMultiEndpointOptions when configuring MultiEndpoints. This name can be used to route an RPC call to this MultiEndpoint by setting the ME_KEY key value of the RPC CallOptions.

GcpMultiEndpointChannel receives a list of GcpMultiEndpointOptions for initial configuration. An updated configuration can be provided at any time later using setMultiEndpoints(List). The first item in the GcpMultiEndpointOptions list defines the default MultiEndpoint that will be used when no MultiEndpoint name is provided with an RPC call.

Example configuration:

• MultiEndpoint named "default" with endpoints:

1. service.example.com:443
2. service-fallback.example.com:443

• MultiEndpoint named "read" with endpoints:

1. ro-service.example.com:443
2. service-fallback.example.com:443
3. service.example.com:443

Let's assume we have a service with read and write operations and the following backends:

• service.example.com -- the main set of backends supporting all operations
• service-fallback.example.com -- read-write replica supporting all operations
• ro-service.example.com -- read-only replica supporting only read operations

With the configuration above GcpMultiEndpointChannel will use the "default" MultiEndpoint by default. It means that RPC calls by default will use the main endpoint and if it is not available then the read-write replica.

To offload some read calls to the read-only replica we can specify "read" MultiEndpoint in the CallOptions. Then these calls will use the read-only replica endpoint and if it is not available then the read-write replica and if it is also not available then the main endpoint.

GcpMultiEndpointChannel creates a GcpManagedChannel channel pool for every unique endpoint. For the example above three channel pools will be created.

[mohanli-ml]

Thanks for the PR! Please take a look at the comments, and let me know if you have any questions.

[wenbozhu]

LG overall.

One thing I was looking for is a way to define a pluggable module to monitor the original endpoint and the underlying mechanism, e.g. /gen204 etc. The implementation will be subject to change.

[nimf]

@wenbozhu I found we don't need to make /gen204 requests as we set up a minimum number of channels to be always connected. This replaces the need of /gen204 because a few HTTP2 connections are always alive to every endpoint. If all connections to an endpoint break, the endpoint is considered unavailable and the minimum number of channels/connections try to reconnect until connection is successful and thus treating the endpoint as available again. Does it make sense?

[wenbozhu]

Are those H2 connections gRPC channels? Do we rely on H2 PING to keep the connections alive, and what's the (default) interval?
Given we will maintain a few (default?) the interval could be increased (maybe by 1/2 of the total number of connections) if we are concerned about the overhead.

[nimf]

Yes those are gRPC channels. We do rely on http2 ping keepalive but if there are no calls then it won’t send pings because keepalive_permit_without_calls is expected to be false. So for the few connections on an unused endpoint no pings will be sent.

[wenbozhu]

Let's discuss this offline. Those channels (to the backup endpoint or to a broken primary endpoint) will not have active requests, so some form of keepalive pings are needed for the purpose of detecting the health of the endpoint.

[rahul2393]

@nimf Any ETA when this will be available in Go https://github.com/GoogleCloudPlatform/grpc-gcp-go.

[nimf]

@rahul2393 preliminary ETA is somewhere in Q1 2023