ca: Don't block server's Stop until handlers finish #1841
Conversation
The ca server's Stop method waits until all running handlers finish. This may deadlock if quorum is not met at the time. Some of the other services like dispatcher used to work like this, but it shouldn't be necessary anymore. Remove the wait group. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
|
LGTM - it seems reasonable to me that we don't have to wait for the handlers to finish - if there is an error seems like the client can just retry and contact another CA server (if it's stopping because it's no longer the leader). Could you explain what causes the deadlock though? The handlers may take a long time to finish because without quorum it cannot read or update data, but does the CA not being able to be stopped prevent quorum from being achieved? Or is it that the manager cannot be stopped because the manager raft server shuts down first, and if quorum is lost then the CA server can't shut down? Or something else entirely? |
|
The deadlock turned out to be a more general problem, and I'm taking some steps to deal with it in #1829. Think of this one as more of a simplification. Basically, you'd call |
Current coverage is 54.48% (diff: 71.42%)@@ master #1841 diff @@
==========================================
Files 102 102
Lines 17025 17018 -7
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
- Hits 9304 9272 -32
- Misses 6585 6603 +18
- Partials 1136 1143 +7
|
|
Ah ok, thanks for explaining! |
|
This looks good to me. |
The ca server's Stop method waits until all running handlers finish.
This may deadlock if quorum is not met at the time. Some of the other
services like dispatcher used to work like this, but it shouldn't be
necessary anymore. Remove the wait group.
cc @cyli @LK4D4