Fix hang when Websocket Read is interrupted

[tuxcanfly]

Updated to catch the quit signal and break out of the loop. Tested that ^C shutsdown btcwallet as expected, with the same env in which it was hanging earlier.

[tuxcanfly]

This helps when the SIGINT is received between messages but not when a message is already being read.

[jrick]

I'll need to take another look at how this is implemented by btcd, but I almost half wonder whether it would makes sense to not increment/decrement the rpcServer WaitGroup for this goroutine to receive the websocket client messages.

If that change is made, then the goroutine that reads from wsc.allRequests (which WebsocketClientRead sends to, and closes when finished) should be switched from a for-range to a for-select to include not just that channel, but also the rpcServer quit chan.

[jrick]

btcd does this the same way as in your PR, so this may be ok, but I'll let @davecgh comment on which design would be better. Personally I think the btcd design can still result in hangs if you time it right.

edit: nvm, I'm wrong. The wait group that btcd increments for the "inHandler" is specific for the client, not the whole server.

[davecgh]

This looks correct to me.

[jrick]

Not sure how it can be. If the server has not begun shutting down (so the select falls through to the default case) and ReadMessage is called and blocks, this can block the shutdown of the entire server until the client disconnects or sends an invalid websocket frame.

[tuxcanfly]

Yes, as I mentioned in the earlier comment, there's still an issue when the timing is right and the server gets blocked on ReadMessage.

[tuxcanfly]

I removed WebsocketClientRead from the waitgroup and it doesn't hang on ReadMessage anymore. But again with the right timing it hangs on RequestHandler, as per the following goroutine stacktrace:

goroutine 50 [chan receive]:
github.com/conformal/btcrpcclient.FutureNotifyReceivedResult.Receive(0x18b122c0, 0x0, 0x0)
        /home/tuxcanfly/Work/golang/src/github.com/conformal/btcrpcclient/notify.go:823 +0x7e
github.com/conformal/btcrpcclient.(*Client).NotifyReceived(0x18cfc5b0, 0xb6291e60, 0x1, 0x1, 0x0, 0x0)
        /home/tuxcanfly/Work/golang/src/github.com/conformal/btcrpcclient/notify.go:901 +0x62
main.(*Account).NewAddress(0x18aee1e0, 0x0, 0x0, 0x0, 0x0)
        /home/tuxcanfly/Work/golang/src/github.com/conformal/btcwallet/account.go:619 +0x3d5
main.GetNewAddress(0xb74eeae0, 0x18e74300, 0x0, 0x0, 0x0, 0x0)
        /home/tuxcanfly/Work/golang/src/github.com/conformal/btcwallet/rpcserver.go:1490 +0x293
main.(*rpcServer).RequestHandler(0x18a7e200)
        /home/tuxcanfly/Work/golang/src/github.com/conformal/btcwallet/rpcserver.go:995 +0xc8
created by main.(*rpcServer).Start
        /home/tuxcanfly/Work/golang/src/github.com/conformal/btcwallet/rpcserver.go:300 +0x84

I tried a similar workaround, i.e. remove RequestHandler from the waitgroup also and it works but not sure if that's the correct way.

[jrick]

Two things:

First, that looks like a btcrpcclient hang.

Second, you modified RequestHandler from a for-select, which reads from s.requests and s.quit, to a for-range that only reads from s.requests. Don't do this. It will now never close, since the s.requests channel is not closed anywhere.

[tuxcanfly]

OK, after removing the RequestHandler from waitgroup I saw that it never received s.quit so I removed it form the for-select. I've reverted the changeset.

I tried that because I was still facing a hang, here's the complete goroutine dump: https://gist.github.com/tuxcanfly/ad54fdab0a90306cf491

[tuxcanfly]

It's a hack but the last commit seems to help, I didn't find any hang on RequestHandler or FutureNotifyReceivedResult in my tests. Tried guarding the client calls with client.Disconnected in the caller itself, but it didn't help so I had to put it in NewAddress.

[tuxcanfly]

As discussed on IRC, there's still one failing case here, although its less frequent:

https://gist.github.com/tuxcanfly/ef0bc10ddebacbfff9f9

[jrick]

I just pushed a btcrpclient fix that causes requests to error when the client has already shutdown, so this check shouldn't be necessary (and isn't really right to begin with, a disconnected client could reconnect and the request would then be reissued).

[tuxcanfly]

Yup, it was a hack. Removed.

[tuxcanfly]

With the latest rpcclient, most hangs seem to have been solved. Only issue I'm seeing is:

goroutine 53 [chan send]:
main.(*rpcServer).RequestHandler(0x18b20360)
    /home/tuxcanfly/Work/golang/src/github.com/conformal/btcwallet/rpcserver.go:1055 +0x240
created by main.(*rpcServer).Start
    /home/tuxcanfly/Work/golang/src/github.com/conformal/btcwallet/rpcserver.go:305 +0x87

which seems to be fixed by:

- r.response <- handlerResponse{result, jsonErr}
+ select {
+ case r.response <- handlerResponse{result, jsonErr}:
+ case <-s.quit:
+ break out
+ }

I guess sending to the response chan was blocked because the receiver had already shutdown.

[jrick]

I would just make this an empty default case and let it fall through, then we don't lose another indent.

[tuxcanfly]

Sorry for nagging but I found at least one instance of a possible hangup. In my tests, this was rare so consider it low priority.

https://gist.github.com/tuxcanfly/629e4b9db61441ffebd6

I tried out a fix by using a goroutine to fetch the response and it seems to help:

tuxcanfly@a439399

[tuxcanfly]

Saw this again today: https://gist.github.com/tuxcanfly/397b365d03664d16f8a3