Fix: lrmd: cancel currently pending STONITH op if stonithd connection is lost #730
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… is lost The currently pending op is moved from rsc->pending_ops to rsc->active (if it is asynchronous). Therefore, that also needs to be cleaned up if the stonithd connection fails. Otherwise, the resource gets stuck forever on an op that will never complete. Example, interacting with the (long fixed) bug mentioned in pull/334: 1. lrmd gets a start action, becomes rsc->active 2. stonithd times out on the action 3. stonithd gets SIGTERMed due to the aforementioned bug 4. lrmd notices and attemps to clean up the pending ops, but misses rsc->active 5. start times out in crmd, attempts to recover 6. lrmd gets a stop action, gets put into rsc->pending_ops 7. lrmd never runs the action since rsc->active is non-NULL 8. stop times out in crmd and the host gets STONITHed due to a failed stop (even though a stonith resource stop is basically a no-op!)
|
@davidvossel fyi |
|
Cancelling in-flight ops seems to be a theme this week. |
beekhof
added a commit
that referenced
this pull request
Jun 12, 2015
Fix: lrmd: cancel currently pending STONITH op if stonithd connection is lost
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This avoids things like nodes getting killed due to hanging stop operations if a start operation caused stonithd to crash.
I saw and debugged this issue on pacemaker 1.1.10+git20130802-1ubuntu2.3 (Ubuntu Trusty). Although the issue manifests itself there due to another bug that is already fixed, it's a bug in its own right, hence this PR. I don't know of any way to trigger this behavior on the current version of pacemaker short of manually killing stonithd in the middle of an op, but it could theoretically happen if stonithd dies for whatever reason.
This is the sequence of events that triggers the bug, conditional on the (long fixed) bug mentioned in PR #334:
(even though a stonith resource stop is basically a no-op!)