Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful shutdown #108

Merged
merged 4 commits into from Jul 8, 2019

Conversation

@algobolson
Copy link
Collaborator

algobolson commented Jul 1, 2019

Summary

Eliminate warning message when a client disconnects normally.

Test Plan

I made a private 3 node network. I watched the logs on the relay node while kill or kill -9 of a leaf node. kill -9 of a node caused a warning message that a node went away unexpectedly, but regular kill exited the node gracefully such that it sent a websocket CloseMessage and the relay node logged no warning.

@algobolson algobolson requested review from zeldovich and tsachiherman Jul 1, 2019
@@ -476,6 +486,7 @@ func (wp *wsPeer) Close() {
atomic.StoreInt32(&wp.didSignalClose, 1)
if atomic.CompareAndSwapInt32(&wp.didInnerClose, 0, 1) {
close(wp.closing)
wp.conn.WriteControl(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""), time.Now().Add(5*time.Second))

This comment has been minimized.

Copy link
@tsachiherman

tsachiherman Jul 1, 2019

Contributor

We need to have this function receive a context, and to have the WriteControl respect that ( somehow ).
We don't want to have a "stuck" endpoint to hang our shutdown process.

sorry - I know it's not easy to implement ;-(

This comment has been minimized.

Copy link
@algobolson

algobolson Jul 1, 2019

Author Collaborator

It has a deadline (now + 5 seconds). How this is called in wsNetwork.go will start a thread per peer, so they'll all wait those 5 seconds in parallel and so at most 5 seconds after WebsocketNetwork.Close() is called it should be done.
I agree in principle, it would be neat to upgrade all of websocket to Write and Read with context. Can we punt for a separate soonish TODO issue?

This comment has been minimized.

Copy link
@tsachiherman

tsachiherman Jul 1, 2019

Contributor

of course. This change is a great improvement on its own and shouldn't be deferred.
We can make it even better tomorrow ;-)

Copy link
Contributor

tsachiherman left a comment

LGTM

Copy link
Contributor

Karmastic left a comment

Want to confirm the answer to my question before this gets checked in

@algobolson algobolson merged commit 5433a1b into algorand:master Jul 8, 2019
2 checks passed
2 checks passed
Travis CI - Pull Request Build Passed
Details
license/cla Contributor License Agreement is signed.
Details
@algobolson algobolson deleted the algobolson:graceful-shutdown branch Jul 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.