Add MaxConnectionAge[Grace] to collector's gRPC server

[jpkrohling]

Closes #1678 by adding two new flags, allowing operators to control the max-connection-age and max-connection-age-grace durations for the collector's gRPC server. Setting the max-connection-age to a few seconds/minutes will cause clients to rediscover collector nodes periodically, by forcing the subchannels to go away after it reaches a certain age. The grace option allows in-flight connections to complete before being forcefully closed.

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

[codecov]

Codecov Report

Merging #3422 (441faaf) into main (62cc3da) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #3422      +/-   ##
==========================================
+ Coverage   96.48%   96.53%   +0.04%     
==========================================
  Files         261      261              
  Lines       15267    15277      +10     
==========================================
+ Hits        14731    14747      +16     
+ Misses        452      448       -4     
+ Partials       84       82       -2     

Impacted Files	Coverage Δ
cmd/collector/app/builder_flags.go	100.00% <100.00%> (ø)
cmd/collector/app/collector.go	75.70% <100.00%> (+0.46%)	⬆️
cmd/collector/app/server/grpc.go	86.36% <100.00%> (+1.36%)	⬆️
plugin/storage/badger/spanstore/reader.go	96.21% <0.00%> (+0.70%)	⬆️
cmd/query/app/static_handler.go	97.60% <0.00%> (+1.79%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62cc3da...441faaf. Read the comment docs.

[pavolloffay]

This is great. Are there any recommended values for Kubernetes environment @jpkrohling ?

[jpkrohling]

This is highly dependent on how elastic the deployment is: the more frequent the changes are, the lower the max-connection-age should be.