-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pings being sent too often by KeepAliveManager #3274
Comments
@dapengzhang0 apologies for the "ping" :-), but you've touched KeepAliveManager and so wanted to get this in front of you (that it's sending pings more often than it should, leading to the server marking the pings as invalid, and shutting down the connection). |
FWIW in KeepAliveManager onTransportIdle, I made this change: if (state == State.PING_SCHEDULED || state == State.PING_DELAYED) {
+ if (pingFuture != null) {
+ pingFuture.cancel(false);
+ } And I now have appropriately-spaced pings and am not getting goaways. Disclaimer I didn't write any tests and there could very likely be a better way of handling this (basically either preventing or correctly handling sendPing getting scheduled twice). |
We purposefully don't cancel the In |
Assigning to @dapengzhang0, because I've got a lot on my plate and @dapengzhang0 is already familiar with the KeepAliveManager. |
If onTransportActive ran while SendPing was already scheduled, we would schedule another SendPing, which seems fine, but the server might observe us sending pings too quickly, and make us GOAWAY. Fixes grpc#3274.
@ejona86 thanks for the feedback! Here's my 2nd attempt, now with a test that was failing pre-fix. Not sure if I want to open this can of worms, but using a mock for the scheduler is kind of unfortunate, given the default mock behavior of In theory |
If onTransportActive ran while SendPing was already scheduled, we would schedule another SendPing, which seems fine, but the server might observe us sending pings too quickly, and make us GOAWAY. Fixes grpc#3274.
If onTransportActive ran while SendPing was already scheduled, we would schedule another SendPing, which seems fine, but the server might observe us sending pings too quickly, and make us GOAWAY. Fixes grpc#3274.
@dapengzhang0 nice, thanks for the link to #2868. That FakeClock/stub executor looks really spiffy. I've used similar approaches in the past, but not that sophisticated. Would be great to see that approach/initial implementation flushed out and end up in something like Guava. |
If onTransportActive ran while SendPing was already scheduled, we would schedule another SendPing, which seems fine, but the server might observe us sending pings too quickly, and make us GOAWAY. Fixes #3274.
Please answer these questions before submitting your issue.
What version of gRPC are you using?
1.5.0
What JVM are you using (
java -version
)?1.8
What did you do?
What did you expect to see?
You can checkout github.com/stephenh/mirror, the latest master has the reproducible error.
Here is the output of some logging that I added to KeepAliveManager:
You can see the 3rd ping is sent only 6 seconds after the 2nd ping.
My guess is that KeepAliveManager is not handling sendPing being scheduled multiple times, and so should check nextKeepaliveTime instead of just the state.
The text was updated successfully, but these errors were encountered: