Skip to content

Conversation

@mattias-wiberg
Copy link
Contributor

Issue

GitHub Issue: #2953
Reporter: @zabranskiy
Package: @grpc/grpc-js
Severity: Bug - Server connections hang indefinitely when clients don't respond to keepalive pings

Root Cause

The server-side keepalive ping timeout handler uses session.close() to terminate connections when clients fail to respond to keepalive pings. However, session.close() performs a graceful shutdown that does not forcibly terminate the underlying connection, causing it to hang indefinitely.

Fix Summary

Changed keepalive error handlers to use session.destroy() instead of session.close() to forcibly close connections when:

  • Keepalive ping timeout expires without response
  • Ping callback receives an error
  • Sending the ping fails

Files Modified

  • packages/grpc-js/src/server.ts (6 occurrences changed)
    • 3 changes in _sessionHandler method (lines ~1609, ~1635, ~1643)
    • 3 changes in _channelzSessionHandler method (lines ~1807, ~1830, ~1843)

Testing Recommendations

To verify the fix:

  1. Start a gRPC server with these options:
    {
      'grpc.keepalive_time_ms': 10000,
      'grpc.keepalive_timeout_ms': 5000,
      'grpc.keepalive_permit_without_calls': 1
    }
  2. Connect a client and send a request
  3. Stop the client process (SIGSTOP) after receiving the first ping response
  4. Observe server logs - after the timeout, the connection should be destroyed (not hanging)

Consistency

This change aligns server-side keepalive handling with the client-side transport implementation (packages/grpc-js/src/transport.ts), which already uses session.destroy() via handleDisconnect() for keepalive timeouts.

Backward Compatibility

This is a bug fix that should not affect normal operation:

  • Proper keepalive implementations will continue to work normally
  • Only affects scenarios where clients fail to respond to keepalive pings
  • Changes do not affect the public API or configuration options

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Oct 24, 2025

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: mattias-wiberg / name: Mattias Wiberg (2901922)

@mattias-wiberg
Copy link
Contributor Author

Bump

@OlegStrokan
Copy link

OlegStrokan commented Nov 28, 2025

@mattias-wiberg I think we’re the only two people on Earth who have discovered this issue. I was trying to implement internet disconnection detection for streaming, using only bare keepalive, without any additional healthcheck/ heartbeat. I even created PR to nest library because it doesn’t map keepalive settings to grpc-js. But later I discovered that the errors still didn’t propagate to the application layer in gprc-js library (I checked with gRPC tracing).
So I ended up generating more CO2 using Claude than all soccer moms combined to understand was ist das
It wasn’t a priority task, so I never proposed a change to the library. Now I see your solution and you’re a lifesaver.
I’m too scared to use a custom grpc-js fork in prod, but it seems like no code review happened. (You’ll probably get a GitHub notification and think “Oh a maintainer finally commented on my pr, but then you will see it’s just some guy with the same issue, sorry 🙃)

@murgatroid99 murgatroid99 changed the base branch from master to @grpc/grpc-js@1.14.x December 1, 2025 15:59
@murgatroid99 murgatroid99 changed the base branch from @grpc/grpc-js@1.14.x to master December 1, 2025 15:59
Copy link
Member

@murgatroid99 murgatroid99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix.

@murgatroid99 murgatroid99 merged commit ec81924 into grpc:master Dec 1, 2025
9 of 10 checks passed
@murgatroid99
Copy link
Member

This is now out in version 1.14.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants