New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature add bad heartbeat logic #22
Feature add bad heartbeat logic #22
Conversation
ACK that I have seen this PR. I will run the heartbeat tests against ActiveMQ, Apollo, and Rabbit. That will take some time, several days at least. With this change ..... I will also test with the race detector. I am a little uneasy about that 'rgr' variable to count sending goroutines. If I have no problems with all the above I will accept this. Otherwise, I'll let you know. If you have some time and this info is available, I would ask that you document here what the negotiated heartbeat values were when that panic occurred. And also what broker was in use. |
Cool, sounds good; thanks for answering back so rapidly Regarding the info requested: I never got printouts of what the values of first and last were when it failed; the steps to repro the bug were to get the service using stompngo running with a healthy rabbit connection, and then filling the hard drive on the rabbit instance such that the GUI isn't reachable/getting 500s. Doing this, eventually (like <30s) the process would error out with the message above. Please let me know if there's additional info that would be helpful |
Thanks for that information. I do not think I will be able to recreate that scenario - fill the hard drive, etc. But this afternoon I ran a full set of heartbeat tests against that code, and saw no ill effects. I want to play a little more with some other informal tests I have, but it looks good at this point. Tests this afternoon were against:
I am going to ignore 'races' for now - when I first wrote stompngo the race detector did not exist. And the entire package needs to be looked at. That is now on my TODO list. I am sure there are races, but I will examine the entire package in different effort. |
Feature add bad heartbeat logic
You should look at the HEAD of the current dev branch. Race conditions uncovered by tests with -race have been eliminated. A number of them were in heartbeat code. Just make sure I have not broken your system with those changes. |
Thanks again for making these changes; I'm not sure that I will be able to load test these changes today due to another part of testing infrastructure being down, but I will do so ASAP and let you know |
No problem, no hurry. I just wanted to let you know that you should probably retest, and report any problems. |
add new logic to deal with panic caused by bad heartbeating