Add basic connection management and keep alive parameters

[vitarb]

Summary:
Added default keep-alive server parameters and enforcement policy for gRPC connection management.
This compliments client-side changes that we recently made in Go and java SDKs.

List of added properties (with frontend.keepAlive prefix) and their default values:

• MinTime: minimum amount of time a client should wait before sending a keepalive ping. Default 1 min. Rationale - set slightly higher than minimal interval to avoid lots of spamming requests.
• PermitWithoutStream: If true, server allows keepalive pings even when there are no active streams(RPCs). Default is true. Rationale - allow clients ping server without having active RPCs.
• MaxConnectionIdle: is a duration for the amount of time after which an idle connection would be closed by sending a GoAway. Default is 10 min. Rationale - default client-side ping interval is 60 sec and long poll interval is every 60-70 sec. If client is idle for 2 minutes it must be dead and connection can be closed.
• MaxConnectionAge: is a duration for the maximum amount of time a connection may exist before it will be closed by sending a GoAway. Default is 20 min. Rationale - periodically close connections to re-balance them more evenly between front-ends.
• MaxConnectionAgeGrace: is an additive period after MaxConnectionAge after which the connection will be forcibly closed. Default is 2 min. Rationale - give some time for the client to finish existing RPCs and then close the channel gracefully.
• Time: After a duration of this time if the server doesn't see any activity it pings the client to see if the transport is still alive. Default is 10 min. Server pings client if it is idle. Rationale - we want to avoid pings to workers as much as possible, setting it reasonably high so we don't hit it often.
• Timeout: After having pinged for keepalive check, the server waits for a duration of Timeout and if no activity is seen even after that the connection is closed. Defaults to 10 sec.

[alexshtin]

My proposal for default values:

• MinTime: 30 sec
• PermitWithoutStream: true
• MaxConnectionIdle: 70 sec (may be just link it to longPollInterval config)
• MaxConnectionAge: 10 min
• MaxConnectionAgeGrace: 70 sec (let ongoing long poll to finish)
• Time: 10 min
• Timeout: 10 sec

[alexshtin]

Can you add comment here saying that this needs to be slightly bigger than long poll interval. And there is another parameter which also correlates with long poll interval.

[vitarb]

Discussed with Alex/Wenquan, and adjusted numbers accordingly.

[vitarb]

Discussed with @mfateev and agreed on a set of parameters that are slightly different from what we've discussed:
List of added properties (with frontend.keepAlive prefix) and their default values:

• MinTime: minimum amount of time a client should wait before sending a keepalive ping. Default 10 sec Rationale - make sure that clients can set minimal value of 10 sec without breaking connections.
• PermitWithoutStream: If true, server allows keepalive pings even when there are no active streams(RPCs). Default is true. Rationale - allow clients ping server without having active RPCs.
• MaxConnectionIdle: is a duration for the amount of time after which an idle connection would be closed by sending a GoAway. Default is 2 min. Rationale - default client-side ping interval is 60 sec and long poll interval is every 60-70 sec. If client is idle for 2 minutes it must be dead and connection can be closed.
• MaxConnectionAge: is a duration for the maximum amount of time a connection may exist before it will be closed by sending a GoAway. Default is 5 min. Rationale - periodically close connections to re-balance them more evenly between front-ends.
• MaxConnectionAgeGrace: is an additive period after MaxConnectionAge after which the connection will be forcibly closed. Default is 70 sec. Rationale - give some time for the client to finish existing RPCs and then close the channel gracefully.
• Time: After a duration of this time if the server doesn't see any activity it pings the client to see if the transport is still alive. Default is 1 min. Server pings client if it is idle. Rationale - if the client is idle for a minute, it's probably not a valid temporal client.
• Timeout: After having pinged for keepalive check, the server waits for a duration of Timeout and if no activity is seen even after that the connection is closed. Defaults to 10 sec.

[wxing1292]

MinTime: minimum amount of time a client should wait before sending a keepalive ping. Default 10 sec Rationale - make sure that clients can set minimal value of 10 sec without breaking connections.

at least this number is too small, let us sync

changing from every 60s to every 10s is 6x increase in amount of load in terms of pinging.
although ping itself is lightweight, it still consume network traffic and requires frontend to respond, meaning health check is not free

[vitarb]

Note that this property defines low end of the range, it doesn't mean we would go as low. In fact I anticipate default on the client side should be 30 sec or 1 min, with a possibility of an override to a lower value, which some OSS clients may want if they would like to detect failures faster and don't have too many workers.

[wxing1292]

comment here specifying that the configs below are intended for small scale use case