Configuration of gRPC MAX_CONNECTION_AGE and MAX_CONNECTION_AGE_GRACE

[slashvar]

🚀 The feature

Expose in the torchserve configuration the following server-side gRPC settings:

• MAX_CONNECTION_AGE : the maximum age of keep alive connections
• MAX_CONNECTION_AGE_GRACE : the grace period to avoid unavailable service when reaching the max connection age.

Both values can be set in the NettyServerBuilder class (see https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html#maxConnectionAge(long,java.util.concurrent.TimeUnit) )

Motivation, pitch

In context of load balanced torchserve instances, keeping connections alive too long will block clients on one instance. Setting the max connection age allow to force client to renew their connections on a regular basis and thus to connect to another instance.

Usage and rational of those settings are described here: https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md

Alternatives

Connection age is not available on client side, thus the only solution is to force a renew of the client connections using a timer.

Additional context

No response

[bjfletcher]

Someone asked a similar question https://discuss.pytorch.org/t/torchserve-grpc-configuration-options-for-client-side-load-balancing/133218