Audit/Fix graceful client shutdown in Netty

[ejona86]

Based on https://groups.google.com/d/topic/grpc-io/6e0GFRSWit8/discussion and the code, it seems that channel.shutdown() when using Netty transport kills open streams.

gracefulClose() calls close() which waits for gracefulShutdownTimeoutMillis. It seems we should either set the grace time to infinity or delay calling close until after all the streams are closed.

We need to investigate why AbstractTransportTest.openStreamPreventsTermination() failed to notice this.

[zpencer]

Looks like the reason AbstractTransportTest failed to catch this problem is that our tests run quickly, thus avoiding the 5 second limit on graceful shutdowns.

Unfortunately the super class Http2ConnectionHandler does not give us an actual way to set the grace time to be infinite. However, the superclass contains everything we need to make the ChannelHandlerContext#close() happen when the last stream is closed. The right place to fix this would be in netty itself. For now, we can set the grace period to be something big enough to be effectively infinite.

[ejona86]

For now, we can set the grace period to be something big enough to be effectively infinite.

Yeah, I'm fine with that approach. Even long-term. 20 years should be enough for anyone!