Permit gPRC clients to send keepalive pings without data #66
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently our router performance degrades significantly when jobs are finishing. After looking into gRPC keepalive settings, I believe one issue is that by default gRPC servers only permit 2 keepalive pings outside the context of data being exchanged. Our Python clients are sending keepalive pings every 5 minutes. Therefore, the servers automatically disconnect clients after 10 minutes. The client does not know this until it tries to make its next call, and errors result. There may be other issues as well, but I would like to test changing this one setting first to observe the effects of setting it.
Helpful documentation:
https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html
https://grpc.io/docs/what-is-grpc/core-concepts/#deadlines
grpc/grpc-java#7237
grpc/grpc#17667