-
Notifications
You must be signed in to change notification settings - Fork 533
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deadlock at disconnect #126
Comments
At first sight it seems worth to remove return at net.go:203 which gives alllogic chance to finish properly without deadlock. But might break some internal usecase |
Could you have a look at #122. I think it's the same issue. Another alternative could be to have a cascading teardown of goroutines when a stop is received by all 4 goroutines being waited on (keepalive optional). |
Close as dup of #122 |
Reopen as seems deadlock still persists on recent codebase |
I have the same problem: hang on with disconnect() method of client.go, at c.workers.Wait(). |
@sdariz if possible for your implementation you can temporary set keep alive to 0 as a workaround |
@troian Thanks, I'm going to try |
Thanks for reporting it again, I think I was getting far to complicated with the keepalive and timers, just trying something much simpler than should be better. |
Cond vars, multiple timers, locks, handling resets. It'd gotten to complex and unwieldy, and worse didn't seem to work reliably. There is now a simple calculation for sending a ping and determining if we haven't received one. There is a side effect that we actually wait the longer PingTimeout or keepalive/2 (when keepalive <= 10 seconds) before triggering that we haven't seen an expected pingresp. Finally, hopefully the resolution to #126
Please test the above, if it's good I'll do a point release on Monday |
looks good for me. At least no hangs at my tests |
Same as troian. No hangs until now. Seems ok |
I currently haven't integrated your latest changes e020008, as there was a national holiday here in India. I'm going to integrate your changes today and report by tomorrow or day-after. I had previously integrated aff1577. I did find some hangs on a long running instance (in the goroutines spawned inside keepalive), but that code doesn't exist in e020008. So, hopefully, there may be no hangs now. |
Any feedback on this @pshirali ? |
I just upgraded to master and I get similar results as @pshirali. We had some behaviour that very much seemed like a deadlock, it always happened after mosquitto would log a socket error and see the client disconnect, but never come back. Ran with master instead of v1.1.0 for the past 24hrs and so far none of it. Interestingly enough the "socket error"s on mosquitto's side seem to also have disappeared, whereas before we'd have about 10 of those over the course of 24hrs. |
I'm going to call this done then, thanks to everyone for the details and feedback you provided. |
Pretty often I see client hangs at disconnect stage
Here is log
You might notice that there is no message done putting puback msg on obound because of the channel oboundP is full and outgoing worker has finished it's job before alllogic.
Quick workaround is to increase channel depth by SetMessageChannelDepth but it's not a case when there is extensive message exchange
The text was updated successfully, but these errors were encountered: