mon: handle monitor lag when killing mgrs #18268
Conversation
Fixes: http://tracker.ceph.com/issues/20629 Signed-off-by: John Spray <john.spray@redhat.com>
| dout(4) << __func__ << ": resetting beacon timeouts due to mon delay " | ||
| "(slow election?) of " << now - last_tick << " seconds" << dendl; | ||
| for (auto &i : last_beacon) { | ||
| i.second = now; |
There was a problem hiding this comment.
i don't think we should be so generous to update all mgr's beacon timestamps. instead, i think we might want to use last_tick + mgr_tick_period - mgr_beacon_grace as the value of cutoff to offset some of the time because of our own fault (slowness).
There was a problem hiding this comment.
Hmm, the idea is that if we have gone slow, then there are probably MgrBeacon messages queued up on peons waiting to be sent through -- the idea of resetting the last_beacons to now is that we will then be giving them mgr_beacon_grace to forward those beacons (assuming they should come soon, now that the mons are back online).
If we used an offset of (now - last_tick) - mgr_tick_period to account for the slowness with a specific number of seconds, then we would only be accounting for that this time through tick(), and we would then be strict again next time through tick -- the effect would be to only allow mgr_tick_period to receive the next beacons, instead of mgr_beacon_grace.
Because we see this behaviour on systems that are slow/laggy, I think it's better to use the more generous approach to allow for peons that are also being slow to forward messages after the cluster recovers (they're swapping a lot or something).
There was a problem hiding this comment.
ahh, right! thanks for your explanation. makes sense to me. the next time when the laggy mon ticks, the innocent mgr(s) won't survive when the fix proposed by me. but they will with your fix.
| dout(4) << __func__ << ": resetting beacon timeouts due to mon delay " | ||
| "(slow election?) of " << now - last_tick << " seconds" << dendl; | ||
| for (auto &i : last_beacon) { | ||
| i.second = now; |
There was a problem hiding this comment.
ahh, right! thanks for your explanation. makes sense to me. the next time when the laggy mon ticks, the innocent mgr(s) won't survive when the fix proposed by me. but they will with your fix.
|
i am adding backport=luminous in the tracker ticket, as i think luminous is also suffering from this problem. |
This is a shameless copy of what MDSMonitor does to handle the same situation.
Fixes: http://tracker.ceph.com/issues/20629
Signed-off-by: John Spray john.spray@redhat.com