Add gRPC MaxConnectionAge & MaxConnectionAgeGrace Options

[krapie]

What this PR does / why we need it:

Add gRPC MaxConnectionAge & MaxConnectionAgeGrace options.

This is to avoid "split-brain" of long-lived gRPC stream connection when performing hash-based load balancing in Yorkie cluster.

For more information, follow: envoyproxy/envoy#26459

gRPC suggest to use MAX_CONNECTION_AGE to avoid long-lived RPC issue on load balancing(https://youtu.be/Naonb2XD_2Q?t=136), but stream is not actually closed even MAX_CONNECTION_AGE is configured.

This is because after MAX_CONNECTION_AGE, server sends GoAway to client, but GoAway is not a signal to close connection instantly. It's just a signal to tell client not to send additional request to server(grpc/grpc-java#8770)

Therefore, MAX_CONNECTION_AGE_GRACE is also introduced to forcibly close stream.

So this is how it works:

1. After stream is established, stream is available for maxConnectionAge.
2. After maxConnectionAge, GoAway frame is sent to client to gracefully close the stream. But stream is not closed at this point.
3. After MaxConnectionAgeGrace, stream is forcibly closed.

Reference: https://pkg.go.dev/google.golang.org/grpc/keepalive

Hence, maxConnectionAge and maxConnectionAgeGrace options are added as gRPC server's keepalive option parameters.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

As I researched some information about long-lived connection, graceful close, etc, I now have good explanation about why close_connections_on_host_set_change(gracefully draining connections) are not working for long-lived gRPC connections.

It is because envoy sends HTTP2 GoAway frame on draining sequence, but GoAway is not for instant connection close. Therefore, no connection close are made after draining, and close_connections_on_host_set_change option will not work as well.

References: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining

Also, I found some interesting issue related to our issue on long-lived connection close: grpc/grpc#26703.

Maybe it will be useful to introduce force_close_connections_on_host_set_change on envoy:: envoyproxy/envoy#26459

Does this PR introduce a user-facing change?:

Additional documentation:

Checklist:

• Added relevant tests or not required
• Didn't break anything

[codecov]

Codecov Report

Merging #512 (20b929b) into main (efe3e87) will increase coverage by 0.84%.
The diff coverage is 23.07%.

@@            Coverage Diff             @@
##             main     #512      +/-   ##
==========================================
+ Coverage   50.22%   51.06%   +0.84%     
==========================================
  Files          64       64              
  Lines        5671     5697      +26     
==========================================
+ Hits         2848     2909      +61     
+ Misses       2481     2429      -52     
- Partials      342      359      +17     

Impacted Files	Coverage Δ
server/config.go	43.33% <0.00%> (-2.02%)	⬇️
server/rpc/config.go	42.85% <0.00%> (-57.15%)	⬇️
server/rpc/server.go	67.79% <60.00%> (-1.60%)	⬇️

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[hackerwins]

Thanks for your contribution.