calling Server.GracefulStop causes "accept tcp host:port: use of closed network connection"

[xh3b4sd]

When starting and stopping the gRPC server gracefully using Server.GracefulStop I noticed the following error.

accept tcp host:port: use of closed network connection

This error does not occur when only calling Server.Stop.

[xh3b4sd]

I see that the example does not even stops the server properly. The onboarding experience with this creates some kind of uncertainty with grpc-go. I am not sure if I am doing something wrong or there is some bug or undocumented desired behaviour in this direction.

See https://github.com/grpc/grpc-go/blob/master/examples/route_guide/server/server.go#L238

[iamqizhao]

This typically means your server has been stopped (e.g.,

grpc-go/server_test.go

Line 63 in 3838369

if got, want := ErrorDesc(err), "use of closed network connection"; !strings.Contains(got, want) {

).

In the example, we do not stop the server for simplicity. In production, server stop code typically stays in the signal handler.

[xh3b4sd]

I do not stop the server before starting it. Why should somebody do that actually? Anyway, the "problem" here is that you stop a server and you get an error no matter what you do. I figured out that it is because of the Close method of https://golang.org/pkg/net/#Listener. Documentation also states this.

// Any blocked Accept operations will be unblocked and return errors.

So my explanation would be that the gRPC server listens and thus calls net.Listener.Accept, which returns always the accept tcp host:port: use of closed network connection error when close is called. gRPC returns this error.

I guess you cannot do anything about this. Anyway it is really odd because users start to ignore errors like this because errors are thrown even nothing was actually wrong. In worst cases this leads to ignoring root causes of serious problems.

[iamqizhao]

we will rework on grpclog so that this kind of error/warning won't be thrown out unless it is under debugging mode.

[toontong]

when the Server.GracefulStop() was celled, all connection from client will be closed, even which was waiting server-side to handling the request. when the server finish the request, it found the tcp-connection was closed. It not Graceful at all. The best way was close the idle connection.

2016/08/31 16:00:12 transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:802->127.0.0.1:44188: use of closed network connection

[iamqizhao]

@toontong In this case, the server sends the response back and finds there is no pending rpc. Now the server closes all its connections, which is the reason this log is generated -- that goroutine is doing a blocking read from the socket which was just closed by the server. GracefulStop is graceful.

[favadi]

What is best way to deal with this error? Should we add a field to Server to make it knows that it has been graceful shutdown? and if the error is "use of closed network connection", we should just ignore it and return nil?

[xh3b4sd]

IMO we should reach out to the go developers and hear what they think. This issue should probably not be solved in grpc-go itself and it should, IMO, definitely not be solved by ignoring the error.

[favadi]

@xh3b4sd can you explain why you think it is a problem with go, not grpc-go?

[xh3b4sd]

In my former comment I pointed out that the error comes from https://golang.org/pkg/net/#Listener, which documentation puts this:

// Any blocked Accept operations will be unblocked and return errors.

So why should something produce an error when I request it to stop any action? Errors indicate faulty behaviour. When I make proper use of the net.Listener API there shouldn't be an error produced. I want to make the listener stop listening. I am not providing wrong arguments. I make proper use of its API. So this is why I think there should be a discussion with the go developers. Otherwise the current behaviour will only lead to people hiding errors, which in the worst case scenario will cause serious harm.

[favadi]

So why should something produce an error when I request it to stop any action?

Maybe I understand something wrongs, but the error returns when we call net.Listener.Accept() after it is closed (with Close() method). I think it is very reasonable, what do you think net.Listener.Accept()` should return in this case instead?

[xh3b4sd]

I understood it this way that you call net.Listener.Accept and each of the blocking accept calls will return an error in case the listener is going to be stopped. This is then not a correct behaviour. In case you call Accept on a listener that is already closed this should return an error, but this is not the case we are talking about as far as I understood.

[hsaliak]

@xh3b4sd do you still feel that this is something that should be solved in grpc-go or are you leaning toward having this resolved in the core net package ? If it's the latter, I'd like to close this issue.

[xh3b4sd]

I do not mind. I am super off this issue at the moment to be honest. If only I have issues, then lets close it. At the moment I am not bothered anymore by it. I think it should be fixed though. I am only not able to invest time into the how and actual doing at the moment. Thanks for asking anyway.

[hsaliak]

I am going to close this issue out for now, if you hit this again and you feel this is something grpc-go should resolve, please reopen. Thanks for filing the issue!